Image pickup apparatus used as action camera

ABSTRACT

An image pickup apparatus capable of eliminating a manual change of an image pickup direction and of easily obtaining an image while focusing attention on experience. A detection unit is worn on a body part other than a head of a user and detects an observation direction of the user. An image pickup unit is worn on the body part and picks up an image. A determination unit determines a recording direction in accordance with the observation direction. An image recording unit records an image corresponding to the recording direction from among a picked-up image. The detection unit and the image pickup unit are located on a user&#39;s median plane and under a user&#39;s jaw in a worn state where the user wears the image pickup apparatus. The detection unit is located at a nearer position to the jaw than the image pickup unit in the worn state.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image pickup apparatus, and inparticular, relates to an image pickup apparatus used as an actioncamera.

Description of the Related Art

When a user picks up an image of an object with a camera, the user needsto continuously direct the camera toward the object. Accordingly, theuser may find it difficult to manage actions other than an image pickupaction because the user is busy in an image pickup operation. Further,the user may find it difficult to focus their attention on theirimmediate surroundings because the user must focus their attention onthe image pickup operation.

For example, if the user is a parent as the user cannot play with achild while performing an image pickup operation with the child as theobject, and the image pickup operation becomes impossible while playingwith the child.

As a further example, if the user performs an image pickup operationwhile watching a sport game, the user cannot focus their attention onthe game (e.g. cheer or remember game contents), and the image pickupoperation becomes impossible while focusing attention to watch thesports game. Similarly, when a user performs an image pickup operationduring group travel, the user cannot focus their attention on the travelexperience to the same extent as other group members, and when the usergives priority to their travel experience, the image pickup operationsuffers as a result.

As a method for solving these matters, methods have been contemplatedwherein a camera is fixed to the head of a user using a fixing-to-headaccessory to pick up an image in an observing direction. This enablesthe user to perform an image pickup operation without being occupiedwith the image pickup operation. Further, there is also contemplated amethod that picks up an image in a wide area with anentire-celestial-sphere camera during experience. This enables a user tofocus attention on their experience during an image pickup operation.After the experience, the user may extract a desired image part frompicked-up entire-celestial-sphere image and edit it to obtain an imageof the experience.

However, these methods need a troublesome action that equips the headwith the fixing-to-head accessory 902 to which a main body of an actioncamera 901 is fixed, as shown in FIG. 34A. Moreover, as shown in FIG.34B, when the user equips the head with the action camera 901 with thefixing-to-head accessory 902, appearance is bad and also a hairstyle ofthe user is disheveled. Furthermore, the user may feel uneasy about theexistence of the fixing-to-head accessory 902 and the action camera 901because of their weights, and may worry about having a bad appearance tothird persons. Accordingly, the user may find it difficult to perform animage pickup operation because the user cannot focus attention on theirexperience in the state shown in FIG. 34B, or because the user feelsresistance to the style shown in FIG. 34B.

In the meantime, the latter method needs series of operations, such asimage conversion and extraction position designation. For example, anentire-celestial-sphere camera 903 equipped with a lens 904 and an imagepickup button 905 as shown in FIG. 35 is known. The lens 904 is one of apair of fish-eye lenses for picking up half-celestial-sphere imagesprovided in both sides of a housing of the entire celestial spherecamera 903. The entire-celestial-sphere camera 903 picks up anentire-celestial-sphere image using these fish-eye lenses. Then, theentire celestial sphere image is obtained by combining projection imagesof the pair of fish-eye lenses.

FIG. 36A, FIG. 36B, and FIG. 36C are views showing examples ofconversion processes of the image picked up by theentire-celestial-sphere camera 903.

FIG. 36A shows an example of the entire-celestial-sphere image obtainedby the entire celestial sphere camera 903, and a user 906, a child 907,and a tree 908 are included as objects. Since this image is anentire-celestial-sphere image obtained by combining projection images ofthe pair of fish-eye lenses, the user 906 is distorted greatly.Moreover, since a body part of the child 907 who is the object that theuser 906 wants to pick up is located in a peripheral part of a pickuparea of the lens 904, the body part distorts greatly in the right andleft directions, and is extended. In the meantime, since the tree 908 isthe object located in front of the lens 904, the tree 908 is picked upwithout great distortion.

In order to generate an image of a visual field at which people areusually looking from the image shown in FIG. 36A, it is necessary toextract a part of the image, to perform plane conversion, and to displaythe converted image.

FIG. 36B is an image located in front of the lens 904 that is extractedfrom the image shown in FIG. 36A. In the image in FIG. 36B, the tree 908is shown in the center in the visual field at which people are usuallylooking. However, since the image in FIG. 36B does not include the child907 who the user 906 wants to pick up, the user has to change anextraction position. Specifically, in this case, it is necessary to movethe extraction position leftward and downward by 30° from the tree 908in FIG. 36A. FIG. 36C shows a displayed image that is obtained byextracting the moved position and by performing the plane conversion. Inthis way, in order to obtain the image in FIG. 36C that the user wantsto pick up from the image in FIG. 36A, the user has to extract anecessary area and has to perform the plane conversion. Accordingly,although the user can focus attention on experience during theexperience (during image pickup), there is a large subsequent workloadfor the user.

Japanese Laid-Open Patent Publication (Kokai) No. 2007-74033 (JP2007-74033A) discloses a technique that uses a second camera that picksup a user in addition to a first camera that picks up an object. Thistechnique calculates a moving direction and visual-line direction of auser from an image picked up by the second camera, determines an imagepickup direction of the first camera, and picks up an image of an objectestimated on the basis of user's viewpoint and state.

Japanese Laid-Open Patent Publication (Kokai) No. 2017-60078 (JP2017-60078A) (Counterpart of US Patent Application 20170085841)discloses an image recording system including a sensor device that isattached to a user's head and an image pickup apparatus that isseparately attached to a user's body or a bag. The sensor deviceconsists of a gyro sensor or an acceleration sensor and detects a user'sobservation direction. The image pickup apparatus picks up an image inthe observation direction detected by the sensor device.

However, since the second camera of JP 2007-74033A picks up an image ofthe user from a position distant from the user, the second camera needshigh optical performance in order to calculate the moving direction andvisual-line direction of the user from the image picked up by the secondcamera. Moreover, since high arithmetic processing capability is neededfor processing the image picked up by the second camera, a scale of anapparatus becomes large. Furthermore, even if the high opticalperformance and the high arithmetic processing capability are satisfied,the user's observation direction cannot be precisely calculated.Accordingly, since an object that the user wants to pick up cannot beestimated with sufficient accuracy on the basis of the user's viewpointand state, an image other than what is wanted by the user may be pickedup.

Moreover, since the sensor device of JP 2017-60078A directly detects auser's observation direction, the user needs to equip the head with thesensor device, which cannot solve troublesomeness in attaching anydevice to the head as mentioned above. Moreover, when the sensor deviceconsists of a gyro sensor or an acceleration sensor, certain accuracycan be obtained in detection of a relative observation direction.However, since accuracy of detection of an absolute observationdirection, especially in the horizontal rotation direction, cannot beobtained, there is an issue in a practical application.

SUMMARY OF THE INVENTION

The present invention provides an image pickup apparatus that is capableof eliminating a manual change of an image pickup direction during animage pickup operation, and that is capable of easily obtaining an imagethat records experience while focusing attention on the experience.

Accordingly, an aspect of the present invention provides an image pickupapparatus including an observation direction detection unit that isadapted to be worn on a body part other than a head of a user and thatis configured to detect an observation direction of the user, at leastone image pickup unit that is adapted to be worn on the body part of theuser and that is configured to pick up an image, a recording directiondetermination unit configured to determine a recording direction inaccordance with the observation direction detected, and an imagerecording unit configured to record an image corresponding to therecording direction determined from among an image picked up by the atleast one image pickup unit. The observation direction detection unitand the image pickup unit are located on a median plane of the user andunder a jaw of the user in a worn state where the user wears the imagepickup apparatus, and the observation direction detection unit islocated at a nearer position to the jaw than the at least one imagepickup unit in the worn state.

According to the present invention, manual change of an image pickupdirection during an image pickup operation becomes unnecessary, and animage that records experience can be easily obtained while focusingattention on the experience.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an external view showing a camera body including animage-pickup/detection unit as an image pickup apparatus according to afirst embodiment.

FIG. 1B is a view showing a state where a user wears the camera body.

FIG. 1C is a view showing a battery unit in the camera body viewed froma rear side in FIG. 1A.

FIG. 1D is an external view showing a display apparatus as a portabledevice according to the first embodiment that is separated from thecamera body.

FIG. 2A is a front view showing the image-pickup/detection unit in thecamera body.

FIG. 2B is a view showing a shape of a band part of a connection memberin the camera body.

FIG. 2C is a rear view showing the image-pickup/detection unit.

FIG. 2D is a top view showing the image-pickup/detection unit.

FIG. 2E is a view showing a configuration of a face direction detectionunit arranged inside the image-pickup/detection unit and under a facedirection detection window in the camera body.

FIG. 2F is a view showing a state where a user wears the camera bodyviewed from a left side of the user.

FIG. 3A, FIG. 3B, and FIG. 3C are views showing details of the batteryunit.

FIG. 4 is a functional block diagram showing the camera body accordingthe first embodiment.

FIG. 5 is a block diagram showing a hardware configuration of the camerabody according to the first embodiment.

FIG. 6 is a block diagram showing a hardware configuration of thedisplay apparatus.

FIG. 7A is a flowchart schematically showing an image pickup/recordingprocess according to the first embodiment executed by the camera bodyand display apparatus.

FIG. 7B is a flowchart showing a subroutine of a preparation process ina step S100 in FIG. 7A according to the first embodiment.

FIG. 7C is a flowchart showing a subroutine of a face directiondetection process in a step S200 in FIG. 7A according to the firstembodiment.

FIG. 7D is a flowchart showing a subroutine of arecording-direction/area determination process in a step S300 in FIG. 7Aaccording to the first embodiment.

FIG. 7E is a flowchart showing a subroutine of a recording-areadevelopment process in a step S500 in FIG. 7A according to the firstembodiment.

FIG. 7F is a view for describing a process in the steps S200 throughS600 in FIG. 7A in a video image mode.

FIG. 8A is a view showing an image of a user viewed from the facedirection detection window.

FIG. 8B is a view showing a case where fluorescent lamps in a roomappear as background in the image of the user viewed from the facedirection detection window.

FIG. 8C is a view showing an image obtained by imaging the user andfluorescent lamps as background shown in FIG. 8B onto a sensor of theinfrared detection device through the face direction detection window ina state where infrared LEDs of the infrared detection device are notlightened.

FIG. 8D is a view showing an image obtained by imaging the user andfluorescent lamps as background shown in FIG. 8B onto the sensor of theinfrared detection device through the face direction detection window ina state where the infrared LEDs are lightened.

FIG. 8E is a view showing a difference image that is calculated bysubtracting the image in FIG. 8C from the image in FIG. 8D.

FIG. 8F is a view showing a result obtained by adjusting shades of thedifference image in FIG. 8E so as to fit with a scale of lightintensities of reflected components of infrared light projected to aface and neck of the user.

FIG. 8G is a view obtained by superimposing reference numerals denotingparts of a user's body, a double circle showing a throat position, and ablack circle showing a chin position on FIG. 8F.

FIG. 8H is a view showing a difference image calculated by the similarmethod as FIG. 8E in directing the user's face to the right.

FIG. 8I is a view showing a result obtained by adjusting shades of thedifference image in FIG. 8H so as to fit with a scale of lightintensities of reflected components of infrared light projected to aface and neck of the user and by superimposing the double circle showingthe throat position and the black circle showing the chin position.

FIG. 8J is a view showing an image of the user who directs the faceupward by 33° viewed from the face direction detection window.

FIG. 8K is a view showing a result obtained by adjusting shades of adifference image, which is calculated by the similar method as FIG. 8Ein a case that the user directs the face upward by 33°, so as to fitwith a scale of light intensities of reflected components of infraredlight projected to a face and neck of the user and by superimposing thedouble circle showing the throat position and the black circle showingthe chin position.

FIG. 9 is a timing chart showing a lighting timing of the infrared LEDsand related signals.

FIG. 10A through FIG. 10D are views describing movements of the user'sface in a vertical direction.

FIG. 11A is a view showing a target visual field set in asuperwide-angle image picked up by an image pickup unit of the camerabody in a case where the user faces the front.

FIG. 11B is a view showing an image in the target visual field extractedfrom the superwide-angle image in FIG. 11A.

FIG. 11C is a view showing the target visual field set in thesuperwide-angle image in a case where the user is observing an A-object.

FIG. 11D is a view showing an image that is obtained by correctingdistortion and blur of an image in the target visual field in FIG. 11Cextracted from the superwide-angle image.

FIG. 11E is a view showing a target visual field set in thesuperwide-angle image in a case where the user is observing the A-objectat a field-angle set value smaller than that in FIG. 11C.

FIG. 11F is a view showing an image that is obtained by correctingdistortion and blur of an image in the target visual field in FIG. 11Eextracted from the superwide-angle image.

FIG. 12A is a view showing an example of the target visual field set inthe superwide-angle image.

FIG. 12B is a view showing an example of the target visual field set inthe superwide-angle image in a case where the field-angle set value isidentical to that of the target visual field in FIG. 12A and where theobservation direction differs.

FIG. 12C is a view showing another example of the target visual fieldset in the superwide-angle image in a case where the field-angle setvalue is identical to that of the target visual field in FIG. 12A andwhere the observation direction differs.

FIG. 12D is a view showing an example of the target visual field set inthe superwide-angle image in a case where the observation direction isidentical to that of the target visual field in FIG. 12C and where thefield-angle set value is smaller.

FIG. 12E is a view showing an example that gives an image stabilizationmargin corresponding to a predetermined image stabilization level aroundthe target visual field shown in FIG. 12A.

FIG. 12F is a view showing an example that gives an image stabilizationmargin corresponding to the same image stabilization level of the imagestabilization margin in FIG. 12E around the target visual field shown inFIG. 12B.

FIG. 12G is a view showing an example that gives an image stabilizationmargin corresponding to the same image stabilization level of the imagestabilization margin in FIG. 12E around the target visual field shown inFIG. 12D.

FIG. 13 is a view showing a menu screen for setting various set valuesof a video image mode that is displayed on a display unit of the displayapparatus before an image pickup operation of the camera body.

FIG. 14 is a flowchart showing a subroutine of a primary recordingprocess in a step S600 in FIG. 7A.

FIG. 15 is a view showing a data structure of an image file generated bythe primary recording process.

FIG. 16 is a flowchart of the subroutine of a transmission process tothe display apparatus in a step S700 in FIG. 7A.

FIG. 17 is a flowchart showing a subroutine of an optical correctionprocess in a step S800 in FIG. 7A.

FIG. 18A through FIG. 18F are views for describing a process of applyingdistortion correction in a step S803 in FIG. 17 .

FIG. 19 is a flowchart showing a subroutine of an image stabilizationprocess in a step S900 in FIG. 7A.

FIG. 20 is an exploded perspective view schematically showing aconfiguration of an image-pickup/detection unit as an image pickupapparatus according to a second embodiment.

FIG. 21 is a view showing a state where a user wears a camera bodyequipped with the image-pickup/detection unit.

FIG. 22A and FIG. 22B are views for describing an operation of a lensbarrier of the camera body.

FIG. 23A is a view showing an internal configuration of theimage-pickup/detection unit in a non-image-pickup state.

FIG. 23B is a view showing the internal configuration of theimage-pickup/detection unit in an image-pickup state.

FIG. 24 is a front view showing the image-pickup/detection unit in astate where a front cover is removed.

FIG. 25 is a front view schematically showing a configuration of animage-pickup/detection unit as an image pickup apparatus according to athird embodiment.

FIG. 26 is an exploded perspective view schematically showing aconfiguration of an image-pickup/detection unit as an image pickupapparatus according to a fourth embodiment.

FIG. 27 is a sectional view along an optical axis OA1 and an opticalaxis OA2 schematically showing an internal configuration of theimage-pickup/detection unit according to the fourth embodiment.

FIG. 28 is a front view schematically showing a configuration of animage-pickup/detection unit as an image pickup apparatus according to afifth embodiment.

FIG. 29 is an exploded perspective view schematically showing aconfiguration of an image-pickup/detection unit as an image pickupapparatus according to a sixth embodiment.

FIG. 30 is a view showing a state where a user wears a camera bodyequipped with the image-pickup/detection unit as the image pickupapparatus according to the sixth embodiment and is performing animage-pickup start operation.

FIG. 31 is a front view showing the image-pickup/detection unitaccording to the sixth embodiment in a state where a front cover isremoved.

FIG. 32A is a view showing examples of shapes of a first button andsecond button of an image-pickup/detection unit as an image pickupapparatus according to a seventh embodiment.

FIG. 32B is a view showing other examples of shapes of the first buttonand second button of the image-pickup/detection unit as the image pickupapparatus according to the seventh embodiment.

FIG. 32C is a view showing still other examples of shapes of the firstbutton and second button of the image-pickup/detection unit as the imagepickup apparatus according to the seventh embodiment.

FIG. 33 is a view showing a state where a user wears a camera bodyequipped with an image-pickup/detection unit as an image pickupapparatus according to an eighth embodiment and is performing animage-pickup start operation.

FIG. 34A and FIG. 34B are views showing a configuration example of acamera fixed to a head using a conventional fixing-to-head accessory.

FIG. 35 is a view showing a configuration example of a conventionalentire-celestial-sphere camera.

FIG. 36A, FIG. 36B, and FIG. 36C are views showing examples ofconversion processes of the image picked up by theentire-celestial-sphere camera in FIG. 35 .

DESCRIPTION OF THE EMBODIMENTS

Hereafter, embodiments according to the present invention will bedescribed in detail by referring to the drawings.

FIG. 1A through FIG. 1D are views for describing a camera systemconsisting of a camera body 1 and a display apparatus 800 that isseparated from the camera body 1. The camera body 1 includes animage-pickup/detection unit 10 as a wearable image pickup apparatusaccording to a first embodiment. Although the camera body 1 and thedisplay apparatus 800 are separated devices in this embodiment, they maybe integrated.

FIG. 1A is an external view showing the camera body 1. The camera body 1is provided with the image-pickup/detection unit 10, a battery unit 90,a right connection member 80R, and a left connection member 80L as shownin FIG. 1A. The right connection member 80R connects theimage-pickup/detection unit 10 and the battery unit 90 on the right sideof a user's body (left side in FIG. 1A). The left connection member 80Lconnects the image-pickup/detection unit 10 and the battery unit 90 onthe left side of the user's body (right side in FIG. 1A).

The image-pickup/detection unit 10 is provided with a face directiondetection window 13, a start switch 14, a stop switch 15, an imagepickup lens 16, an LED 17, and microphones 19L and 19R.

The face direction detection window 13 permits transmission of infraredlight projected from infrared LEDs 22 (FIG. 5 ) built in theimage-pickup/detection unit 10 to detect positions of face parts of theuser. The face direction detection window 13 also permits transmissionof reflected infrared light from the face.

The start switch 14 is used to start an image pickup operation. The stopswitch 15 is used to stop the image pickup operation. The image pickuplens 16 guides light to be picked up to a solid state image sensor 42(FIG. 5 ) inside the image-pickup/detection unit 10. The LED 17indicates a state that the image pickup operation is on-going.Additionally or alternatively, the LED 17 can function as a warninglight.

The microphones 19R and 19L take in peripheral sound. The microphone 19Ltakes in sound of the left side of user's periphery (right side in FIG.1A). The microphone 19R takes in sound of the right side of the user'speriphery (left side in FIG. 1A).

FIG. 1B is a view showing a state where the user wears the camera

When the user wears the camera body 1 so that the battery unit 90 islocated proximate to a user's back side and the image-pickup/detectionunit 10 is located proximate to the front side of the user's body, theimage-pickup/detection unit 10 is supported while being energized in adirection toward a chest by the left and right connection members 80Land 80R that are respectively connected to the left and right ends ofthe image-pickup/detection unit 10. Thereby, the image-pickup/detectionunit 10 is positioned in front of clavicles of the user. At this time,the face direction detection window 13 is located under a jaw of theuser. An infrared condenser lens 26 shown in FIG. 2E mentioned later isarranged inside the face direction detection window 13. An optical axis(detection optical axis) of the infrared condenser lens 26 is directedto the user's face and is directed to a different direction from anoptical axis (image pickup optical axis) of the image pickup lens 16. Aface direction detection unit 20 (see FIG. 5 ) including the infraredcondenser lens 26 detects a user's observation direction on the basis ofthe positions of face parts. This enables an image pickup unit 40mentioned later to pick up an image of an object in the observationdirection. Adjustment of the setting position due to individualdifference of a body shape and difference in clothes will be mentionedlater.

Moreover, since the image-pickup/detection unit 10 is arranged in thefront side of the body and the battery unit 90 is arranged in the backface in this way, weight of the camera body 1 is distributed, whichreduces user's fatigue and reduces displacement of the camera body 1 dueto forces on the camera body 1 caused by movement of the user.

Although the example in which the user wears the camera body 1 so thatthe image-pickup/detection unit 10 will be located in front of theclavicles of the user is described in this embodiment, this example isnot imperative. That is, the user may wear the camera body 1 in anyposition of the user's body other than the head as long as the camerabody 1 can detect the user's observation direction and the image pickupunit 40 can pick up an image of an object in the observation direction.

FIG. 1C is a view showing the battery unit 90 viewed from a rear side inFIG. 1A. The battery unit 90 is provided with a charge cable insertingslot 91, adjustment buttons 92L and 92R, and a backbone escape cutout 93as shown in FIG. 1C.

A charge cable (not shown) can be connected to the charge cableinserting slot 91. An external power source charges internal batteries94L and 94R (see FIG. 3A) and supplies electric power to theimage-pickup/detection unit 10 through the charge cable.

Adjustment buttons 92L and 92R are used to adjust the respective lengthsof the band parts 82L and 82R of the left and right connection members80L and 80R. The adjustment button 92L is used to adjust the left bandpart 82L, and the adjustment button 92R is used to adjust the right bandpart 82R. Although the lengths of the band parts 82L and 82R areindependently adjusted with the adjustment buttons 92L and 92R in theembodiment, the lengths of the band parts 82L and 82R may besimultaneously adjusted with one button.

The backbone escape cutout 93 is formed by shaping the battery unit 90so that the battery unit 90 will not touch the backbone. Since thebackbone escape cutout 93 avoids a convex part of the backbone of thebody, displeasure of wearing is reduced and lateral displacement of thebattery unit 90 is prevented.

FIG. 1D is an external view showing the display apparatus 800 as aportable device according to the first embodiment that is separated fromthe camera body 1. As shown in FIG. 1D, the display apparatus 800 isprovided with an A-button 802, a display unit 803, a B-button 804, anin-camera 805, a face sensor 806, an angular speed sensor 807, and anacceleration sensor 808. Moreover, the display apparatus 800 is providedwith a wireless LAN unit (not shown in FIG. 1D) that enables high-speedconnection with the camera body 1.

The A-button 802 has a function of a power button of the displayapparatus 800. The display apparatus 800 receives an ON/OFF operation bya long press of the A-button 802 and receives a designation of anotherprocess timing by a short press of the A-button 802.

The display unit 803 is used to check an image picked up by the camerabody 1 and can display a menu screen required for setting. In thisembodiment, a transparent touch sensor that is provided on the surfaceof the display unit 803 receives a touch operation to a screen (forexample, a menu screen) that is displaying.

The B-button 804 functions as a calibration button 854 used for acalibration process mentioned later. The in-camera 805 can pick up animage of a person who is observing the display apparatus 800.

The face sensor 806 detects a face shape and an observation direction ofthe person who is observing the display apparatus 800. A concreteconfiguration of the face sensor 806 is not limited. For example, astructural optical sensor, a ToF (Time of Flight) sensor, and amillimeter-wave radar may be employed.

Since the angular speed sensor 807 is built in the display apparatus800, it is shown by a dotted line as a meaning of a perspective view.Since the display apparatus 800 of this embodiment is also provided witha function of the calibrator mentioned later, a triaxial gyro sensorthat enables detection in X, Y, and Z directions is mounted. Theacceleration sensor 808 detects a posture of the display apparatus 800.

It should be noted that a general smart phone is employed as the displayapparatus 800 according to this embodiment. The camera system of theembodiment is achieved by matching firmware in the smart phone tofirmware of the camera body 1. In the meantime, the camera system of theembodiment can be achieved by matching the firmware of the camera body 1to an application and OS of the smart phone as the display apparatus800.

FIG. 2A through FIG. 2F are views describing the image-pickup/detectionunit 10 in detail. In views from FIG. 2A, a component that has the samefunction of a part that has been already described is indicated by thesame reference numeral and its description in this specification isomitted.

FIG. 2A is a front view showing the image-pickup/detection unit 10.

The right connection member 80R has the band part 82R and anangle-holding member 81R of hard material that holds an angle withrespect to the image-pickup/detection unit 10. The left connectionmember 80L has the band part 82L and an angle-holding member 81Lsimilarly.

FIG. 2B is a view showing the shapes of the band parts 82L and 82R ofthe left and right connection members 80L and 80R. In FIG. 2B, the angleholding members 81L and 81R are shown as transparent members in order toshow the shapes of the band parts 82L and 82R.

The band part 82L is provided with a left connecting surface 83L and anelectric cable 84 that are arranged at the left side of the user's body(right side in FIG. 2B) when the user wears the camera body 1. The bandpart 82R is provided with a right connecting surface 83R arranged at theright side of the user's body (left side in FIG. 2B) when the user wearsthe camera body 1.

The left connecting surface 83L is connected with the angle holdingmember 81L, and its sectional shape is an ellipse but is not a perfectcircle. The right connecting surface 83R also has a similar ellipticalshape. The right connecting surface 83R and left connecting surface 83Lare arranged bisymmetrically in a reverse V-shape. That is, the distancebetween the right connecting surface 83R and the left connecting surface83L becomes shorter toward the upper side from the lower side in FIG.2B. Thereby, since the long axis directions of the left and rightconnecting surfaces 83L and 83R match the user's body when the userhangs the camera body 1, the band parts 82L and 82R touch the user'sbody comfortably and movement of the image-pickup/detection unit 10 inthe left-and-right direction and front-and-back direction can beprevented.

The electric cable 84 is wired inside the band part 82L and electricallyconnects the battery unit 90 and the image-pickup/detection unit 10. Theelectric cable 84 connects the power source of the battery unit 90 tothe image-pickup/detection unit 10 or transfers an electrical signalwith an external apparatus.

FIG. 2C is a rear view showing the image-pickup/detection unit 10. FIG.2C shows the side that contacts to the user's body. That is, FIG. 2C isa view viewed from the opposite side of FIG. 2A. Accordingly, thepositional relationship between the right connection member 80R and theleft connection member 80L is reverse to FIG. 2A.

The image-pickup/detection unit 10 is provided with a power switch 11,an image pickup mode switch 12, and chest contact pads 18 a and 18 b atthe back side. The power switch 11 is used to switch ON/OFF of the powerof the camera body 1. Although the power switch 11 of this embodiment isa slide lever type, it is not limited to this. For example, the powerswitch 11 may be a push type switch or may be a switch that isintegrally constituted with a slide cover (not shown) of the imagepickup lens 16.

The image pickup mode switch 12 is used to change an image pickup modeand can change a mode about an image pickup operation. In thisembodiment, the image pickup mode switch 12 can select the image pickupmode from among a still image mode, a video image mode, and abelow-mentioned preset mode that is set using the display apparatus 800.In this embodiment, the image pickup mode switch 12 is a slide leverswitch that can select one of “Photo”, “Normal”, and “Pre” shown in FIG.2C. The image pickup mode shifts to the still image mode by sliding to“Photo”, shifts to the video image mode by sliding to “Normal”, andshifts to the preset mode by sliding to “Pre”. It should be noted thatthe configuration of the image pickup mode switch 12 is not limited tothe embodiment as long as the switch can change the image pickup mode.For example, the image pickup mode switch 12 may consist of threebuttons of “Photo”, “Normal”, and “Pre”.

The chest contact pads 18 a and 18 b touch the user's body when theimage-pickup/detection unit 10 is energized. As shown in FIG. 2A, theimage-pickup/detection unit 10 is formed so that a lateral(left-and-right) overall length will become longer than a vertical(up-and-down) overall length in wearing the camera body 1. The chestcontact pads 18 a and 18 b are respectively arranged in vicinities ofright and left ends of the image-pickup/detection unit 10. Thisarrangement reduces rotational blur in the left-and-right directionduring the image pickup operation of the camera body 1. Moreover, thechest contact pads 18 a and 18 b prevent the power switch 11 and theimage pickup mode switch 12 from touching the user's body. Furthermore,the chest contact pads 18 a and 18 b prevent heat transmission to theuser's body even if the image-pickup/detection unit 10 heats up due to along-time image pickup operation and are used for the adjustment of theangle of the image-pickup/detection unit 10.

FIG. 2D is a top view showing the image-pickup/detection unit 10. Asshown in FIG. 2D, the face direction detection window 13 is provided inthe central part of the top surface of the image-pickup/detection unit10, and the chest contact pads 18 a and 18 b are projected from theimage-pickup/detection unit 10.

FIG. 2E is a view showing a configuration of the face directiondetection unit 20 arranged inside the image-pickup/detection unit 10 andunder the face direction detection window 13. The face directiondetection unit 20 is provided with the infrared LEDs 22 and the infraredcondenser lens 26. The face direction detection unit 20 is also providedwith an infrared LED lighting circuit 21 and an infrared detectiondevice 27 shown in FIG. 5 mentioned later.

The infrared LEDs 22 project infrared light 23 (FIG. 5 ) toward theuser. The infrared condenser lens 26 images reflected light 25 (FIG. 5 )from the user in projecting the infrared light 23 from the infrared LEDs22 onto a sensor (not shown) of the infrared detection device 27.

FIG. 2F is a view showing a state where a user wears the camera body 1viewed from the left side of the user.

An angle adjustment button 85L is provided in the angle holding member81L and is used in adjusting the angle of the image-pickup/detectionunit 10. An angle adjustment button (not shown in FIG. 2F) is providedin the opposite angle holding member 81R in the symmetrical position ofthe angle adjustment button 85L. Although the angle adjustment buttonsare actually visible in FIG. 2A, FIG. 2C, and FIG. 2D, they are omittedto simplify the description.

When moving the angle holding member 81L upwardly or downwardly in FIG.2F while pressing the angle adjustment button 85L, the user can changethe angle between the image-pickup/detection unit 10 and the angleholding member 81L. The right side is the same as the left side.Moreover, projection angles of the chest contact pads 18 a and 18 b canbe changed. The functions of these two kinds of angle change members(the angle adjustment buttons and chest contact pads) can adjust theimage-pickup/detection unit 10 so as to keep the optical axis of theimage pickup lens 16 horizontally irrespective of individual differenceof a chest position shape.

FIG. 3A, FIG. 3B, and FIG. 3C are views showing details of the batteryunit 90. FIG. 3A is a partially transparent back view showing thebattery unit 90.

As shown in FIG. 3A, the left battery 94L and right battery 94R aresymmetrically mounted inside the battery unit 90 in order to keep weightbalance. In this way, since the left and right batteries 94L and 94R arearranged symmetrically with the central part of the battery unit 90, theweight balance in the left-and-right direction is achieved and theposition displacement of the camera body 1 is prevented. It should benoted that the battery unit 90 may mount a single battery.

FIG. 3B is a top view showing the battery unit 90. The batteries 94L and94R are shown as the transparent members also in FIG. 3B. As shown inFIG. 3B, since the batteries 94L and 94R are symmetrically arranged atboth the sides of the backbone escape cutout 93, the user can wear thebattery unit 90 that is relatively heavy without any burden.

FIG. 3C is a rear view showing the battery unit 90. FIG. 3C is the viewviewed from the side touched to the user's body, i.e., is the viewviewed from the opposite side of FIG. 3A. As shown in FIG. 3C, thebackbone escape cutout 93 is provided in the center along the backboneof the user.

FIG. 4 is a functional block diagram showing the camera body 1.Hereinafter, the process executed by the camera body 1 will be describedroughly using FIG. 4 . Details will be mentioned later.

As shown in FIG. 4 , the camera body 1 is provided with the facedirection detection unit 20, a recording-direction/field-angledetermination unit 30, the image pickup unit 40, an imageextraction/development unit 50, a primary recording unit 60, atransmission unit 70, and a second controller 111. These functionalblocks are achieved by control of an overall control CPU 101 (FIG. 5 )that controls the entire camera body 1.

The face direction detection unit 20 (an observation direction detectionunit) is a functional block executed by the above-mentioned infraredLEDs 22, the infrared detection device 27, etc. The face directiondetection unit 20 estimates an observation direction by detecting theface direction and passes the observation direction to therecording-direction/field-angle determination unit 30.

The recording-direction/field-angle determination unit (a recordingdirection determination unit) 30 determines information about a positionand an area that will be extracted from an image picked up by the imagepickup unit 40 by performing various calculations on the basis of theobservation direction estimated by the face direction detection unit 20.And then, the information is passed to the image extraction/developmentunit 50. The image pickup unit 40 converts light from an object into awide-angle image and passes the image to the imageextraction/development unit 50.

The image extraction/development unit 50 extracts an image that the userlooks at from the image passed from the image pickup unit 40 by usingthe information passed from the recording-direction/field-angledetermination unit 30. Then, the image extraction/development unit 50develops the extracted image and passes the developed image to theprimary recording unit 60.

The primary recording unit 60 is a functional block constituted by aprimary memory 103 (FIG. 5 ) etc., records image information, and passesthe image information to the transmission unit 70 at a required timing.

The transmission unit 70 is wirelessly connected with predeterminedcommunication parties, such as the display apparatus 800 (FIG. 1D), acalibrator 850, and a simplified display apparatus 900, and communicateswith these parties.

The display apparatus 800 is connectable to the transmission unit 70through a high-speed wireless LAN (hereinafter referred to as a“high-speed wireless network”). In this embodiment, the high-speedwireless network employs wireless communication corresponding to theIEEE802.11ax (WiFi 6) standard. In the meantime, wireless communicationcorresponding to other standards, such as the WiFi 4 standard and theWiFi 5 standard, may be employed. Moreover, the display apparatus 800may be a dedicated apparatus developed for the camera body 1 or may be ageneral smart phone, a tablet terminal, etc.

In addition, the display apparatus 800 may be connected to thetransmission unit 70 through a small-power wireless network, may beconnected through both the high-speed wireless network and small-powerwireless network, or may be connected while switching the networks. Inthis embodiment, large amount data like an image file of a video imagementioned later is transmitted through the high-speed wireless network,and small amount data and data that does not need quick transmission aretransmitted through the small-power wireless network. Although theBluetooth is used for the small-power wireless network in thisembodiment, other short-distance wireless communications, such as theNFC (Near Field Communication), may be employed.

The calibrator 850 performs initial setting and individual setting ofthe camera body 1, and is connectable to the transmission unit 70through the high-speed wireless network in the same manner as thedisplay apparatus 800. Details of the calibrator 850 are mentionedlater. Moreover, the display apparatus 800 may have the function of thecalibrator 850.

The simplified display apparatus 900 is connectable to the transmissionunit 70 only through the small-power wireless network, for example.Although the simplified display apparatus 900 cannot performcommunication of a video image with the transmission unit 70 due to timerestriction, it can transmit an image pickup start/stop timing and canbe used for an image check of a composition check level. Moreover, thesimplified display apparatus 900 may be a dedicated apparatus developedfor the camera body 1 as well as the display apparatus 800 or may be asmart watch etc.

FIG. 5 is a block diagram showing a hardware configuration of the camerabody 1. Moreover, the configurations and functions described using FIG.1A through FIG. 1C are indicated by the same reference numerals andtheir detailed descriptions will be omitted.

As shown in FIG. 5 , the camera body 1 is provided with the overallcontrol CPU 101, power switch 11, image pickup mode switch 12, facedirection detection window 13, start switch 14, stop switch 15, imagepickup lens 16, and LED 17.

The camera body 1 is further provided with the infrared LED lightingcircuit 21, infrared LEDs 22, infrared condenser lens 26, and infrareddetection device 27 that constitute the face direction detection unit 20(FIG. 4 ).

Moreover, the camera body 1 is provided with the image pickup unit 40(FIG. 4 ), which consists of an image pickup driver 41, a solid stateimage sensor 42, and an image signal processing circuit 43, and thetransmission unit 70 (FIG. 4 ), which consists of a small-power wirelesscommunication unit 71 and high-speed wireless communication unit 72.

Although the camera body 1 has the single image pickup unit 40 in thisembodiment, it may have two or more image pickup units in order to pickup a 3D image, to pick up an image of which a field angle is wider thanan image obtained by a single image pickup unit, or to pick up images indifferent directions.

The camera body 1 is provided with various memories, such as alarge-capacity nonvolatile memory 51, an internal nonvolatile memory102, the primary memory 103, etc.

Furthermore, the camera body 1 is provided with an audio processor 104,a speaker 105, a vibrator 106, an angular speed sensor 107, anacceleration sensor 108, and various switches 110.

The switches like the power switch 11, which are described above usingFIG. 2C, are connected to the overall control CPU 101. The overallcontrol CPU 101 controls the entire camera body 1. Therecording-direction/field-angle determination unit 30, imageextraction/development unit 50, and second controller 111 in FIG. 4 areachieved by the overall control CPU 101.

The infrared LED lighting circuit 21 controls lighting of the infraredLEDs 22 (FIG. 2E) to control projection of the infrared light 23directed to the user from the infrared LEDs 22.

The face direction detection window 13 is constituted by a visible lightcut filter that hardly permits transmission of visible light andsufficiently permits transmission of the infrared light 23 and itsreflected light 25 that belong to infrared region. The infraredcondenser lens 26 condenses the reflected light 25.

The infrared detection device 27 has a sensor that detects the reflectedlight 25 condensed by the infrared condenser lens 26. The sensorconverts an image formed by the condensed reflected light 25 into sensordata and passes the sensor data to the overall control CPU 101.

As shown in FIG. 1B, when the user wears the camera body 1, the facedirection detection window 13 is located under a user's jaw.Accordingly, as shown in FIG. 5 , the infrared light 23 projected fromthe infrared LEDs 22 transmits the face direction detection window 13and irradiates an infrared irradiation surface 24 near the user's jaw.Moreover, the reflected light 25 reflected from the infrared irradiationsurface 24 transmits the face direction detection window 13 and iscondensed by the infrared condenser lens 26 onto the sensor in theinfrared detection device 27.

The various switches 110 are not shown in FIG. 1A through FIG. 1C. Thevarious switches 110 are used to execute functions that are unrelated tothis embodiment.

The image pickup driver 41 includes a timing generator etc., generatesvarious timing signals, outputs the timing signals to sections relatedto the image pickup operation, and drives the solid state image sensor42.

The solid state image sensor 42 outputs the signal obtained byphotoelectric conversion of the object image formed through the imagepickup lens 16 described using FIG. 1A to the image signal processingcircuit 43.

The image signal processing circuit 43 outputs picked-up image data,which is generated by applying a clamp process and an A/D conversionprocess, etc. to the signal from the solid state image sensor 42, to theoverall control CPU 101.

The internal nonvolatile memory 102 is constituted by a flash memoryetc. and stores a boot program of the overall control CPU 101 and setvalues of various program modes. In this embodiment, a set value of anobservation visual field (field angle) and a set value of an effectlevel of an image stabilization process are recorded.

The primary memory 103 is constituted by a RAM etc. and temporarilystores processing image data and a calculation result of the overallcontrol CPU 101.

The large-capacity nonvolatile memory 51 stores image data. In thisembodiment, the large-capacity nonvolatile memory 51 is a semiconductormemory that is not detachable. However, the large-capacity nonvolatilememory 51 may be constituted by a detachable storage medium like an SDcard, and may be used together with the internal nonvolatile memory 102.

The small-power wireless communication unit 71 exchanges data with thedisplay apparatus 800, the calibrator 850, and the simplified displayapparatus 900 through the small-power wireless network. The high-speedwireless communication unit 72 exchanges data with the display apparatus800 and the calibrator 850 through the high-speed wireless network.

The audio processor 104 processes outside sound (analog signals)collected by the microphones 19L and 19R and generates an audio signal.

In order to notify the user of a state of the camera body 1 and to warnthe user, the LED 17 emits light, the speaker 105 outputs sound, and thevibrator 106 vibrates.

The angular speed sensor 107 uses a gyro etc. and detects movement ofthe camera body 1 itself as gyro data. The acceleration sensor 108detects the posture of the image-pickup/detection unit 10.

FIG. 6 is a block diagram showing a hardware configuration of thedisplay apparatus 800. The components that have been described usingFIG. 1D are indicated by the same reference numerals and theirdescriptions will be omitted to simplify the description.

As shown in FIG. 6 , the display apparatus 800 is provided with adisplay-apparatus controller 801, the A-button 802, the display unit803, the B-button 804, the face sensor 806, the angular speed sensor807, the acceleration sensor 808, an image signal processing circuit809, and various switches 811.

Moreover, the display apparatus 800 is provided with an internalnonvolatile memory 812, a primary memory 813, a large-capacitynonvolatile memory 814, a speaker 815, a vibrator 816, an LED 817, anaudio processor 820, a small-power wireless communication unit 871, anda high-speed wireless communication unit 872. The above-mentionedcomponents are connected to the display-apparatus controller 801. Thedisplay-apparatus controller 801 is constituted by a CPU and controlsthe display apparatus 800.

The image signal processing circuit 809 bears equivalent functions withthe image pickup driver 41, solid state image sensor 42, and imagesignal processing circuit 43 inside the camera body 1. The image signalprocessing circuit 809 constitutes the in-camera 805 in FIG. 1D togetherwith an in-camera lens 805 a. The display-apparatus controller 801processes the data output from the image signal processing circuit 809.The contents of the process of the data will be mentioned later.

The various switches 811 are used to execute functions that areunrelated to this embodiment. The angular speed sensor 807 uses a gyroetc. and detects movement of the display apparatus 800.

The acceleration sensor 808 detects a posture of the display apparatus800 itself. The angular speed sensor 807 and the acceleration sensor 808are built in the display apparatus 800, and respectively have thefunctions equivalent to that of the above-mentioned angular speed sensor107 and acceleration sensor 108 of the camera body 1.

The internal nonvolatile memory 812 is constituted by a flash memoryetc. and stores a boot program of the display-apparatus controller 801and set values of various program modes.

The primary memory 813 is constituted by a RAM etc. and temporarilystores processing image data and a calculation result of the imagesignal processing circuit 809. In this embodiment, when a video image isrecording, gyro data detected with the angular speed sensor 107 atpickup time of each frame is stored into the primary memory 813 inassociation with the frame.

The large-capacity nonvolatile memory 814 stores image data of thedisplay apparatus 800. In this embodiment, the large-capacitynonvolatile memory 814 is constituted by a detachable memory like an SDcard. It should be noted that the large-capacity nonvolatile memory 814may be constituted by a fixed memory as with the large-capacitynonvolatile memory 51 in the camera body 1.

In order to notify the user of a state of the display apparatus 800 andto warn the user, the speaker 815 outputs sound, the vibrator 816vibrates, and the LED 817 emits light.

The audio processor 820 processes outside sound (analog signals)collected by the left microphone 819L and right microphone 819R andgenerates an audio signal.

The small-power wireless communication unit 871 exchanges data with thecamera body 1 through the small-power wireless network. The high-speedwireless communication unit 872 exchanges data with the camera body 1through the high-speed wireless network.

The face sensor 806 is provided with an infrared LED lighting circuit821 and infrared LEDs 822, an infrared condenser lens 826, and aninfrared detection device 827.

The infrared LED lighting circuit 821 has the function equivalent tothat of the infrared LED lighting circuit 21 in FIG. 5 and controlslighting of the infrared LEDs 822 to control projection of the infraredlight 823 directed to the user from the infrared LEDs 822. The infraredcondenser lens 826 condenses the reflected light 825 of the infraredlight 823.

The infrared detection device 827 has a sensor that detects thereflected light 825 condensed by the infrared condenser lens 826. Thesensor converts the condensed reflected light 825 into sensor data andpasses the sensor data to the display-apparatus controller 801.

When the face sensor 806 shown in FIG. 1D is directed to the user, aninfrared irradiation surface 824 that is the entire face of the user isirradiated with the infrared light 823 projected from the infrared LEDs822 as shown in FIG. 6 . Moreover, the reflected light 825 reflectedfrom the infrared irradiation surface 824 is condensed by the infraredcondenser lens 826 onto the sensor in the detection device 827.

Other functions 830 are functions of a smart phone, such as a telephonefunction, that are not related to the embodiment.

Hereinafter, how to use the camera body 1 and display apparatus 800 willbe described. FIG. 7A is a flowchart schematically showing an imagepickup/recording process according to the first embodiment executed bythe camera body 1 and display apparatus 800.

In order to assist the description, a reference numeral shown in FIG. 4and FIG. 5 of a unit that executes a process in each step is shown on aright side of each step in FIG. 7A. That is, steps S100 through S700 inFIG. 7A are executed by the camera body 1, and steps S800 through S1000in FIG. 7A are executed by the display apparatus 800.

When the power switch 11 is set to ON and power of the camera body 1turns ON, the overall control CPU 101 is activated and reads the bootprogram from the internal nonvolatile memory 102. After that, in thestep S100, the overall control CPU 101 executes a preparation processthat performs setting of the camera body 1 before an image pickupoperation. Details of the preparation process will be mentioned laterusing FIG. 7B.

In a step S200, the face direction detection process that estimates anobservation direction based on a face direction detected by the facedirection detection unit 20 is executed. Details of the face directiondetection process will be mentioned later using FIG. 7C. This process isexecuted at a predetermined frame rate. In a step S300, therecording-direction/field-angle determination unit 30 executes arecording-direction/area determination process. Details of therecording-direction/area determination process will be mentioned laterusing FIG. 7D.

In a step S400, the image pickup unit 40 picks up an image and generatespickup image data. In a step S500, the image extraction/development unit50 extracts an image from the pickup image data generated in the stepS400 according to recording-direction/field-angle information determinedin the step S300 and performs a recording area development process thatdevelops the extracted area. Details of the recording area developmentprocess will be mentioned later using FIG. 7E.

In a step S600, the primary recording unit (an image recording unit) 60executes a primary recording process that stores the image developed inthe step S500 into the primary memory 103 as image data. Details of theprimary recording process will be mentioned later using FIG. 14 .

In the step S700, the transmission unit 70 executes a transmissionprocess to the display apparatus 800 that wirelessly transmits the imageprimarily recorded in the step S600 to the display apparatus 800 at adesignated timing. Details of the transmission process to the displayapparatus 800 will be mentioned later using FIG. 16 .

The steps from the step S800 are executed by the display apparatus 800.In the step S800, the display-apparatus controller 801 executes anoptical correction process that corrects optical aberration of the imagetransferred from the camera body 1 in the step S700. Details of theoptical correction process will be mentioned later using FIG. 17 .

In a step S900, the display-apparatus controller 801 applies the imagestabilization process to the image of which the optical aberration hasbeen corrected in the step S800. Details of the image stabilizationprocess will be mentioned later using FIG. 19 . It should be noted thatthe order of the step S800 and the step S900 may be inverted. That is,the image stabilization process may be executed in advance and theoptical correction process may be executed after that.

The display-apparatus controller 801 executes a secondary recordingprocess that records the image to which the optical correction processin the step S800 and the image stabilization process in the step S900have been applied into the large-capacity nonvolatile memory 814 in thestep S1000, and then, finishes this process.

Next, the subroutines in the respective steps in FIG. 7A will bedescribed in detail using FIG. 7B through FIG. 7F and other drawings inthe order of the processes. FIG. 7B is a flowchart showing thesubroutine of the preparation process in the step S100 in FIG. 7A.Hereinafter, this process is described using the components shown inFIG. 2A through FIG. 2F and FIG. 5 .

It is determined whether the power switch 11 is ON in a step S101. Theprocess waits when the power is OFF. When the power becomes ON, theprocess proceeds to a step S102.

In the step S102, the mode selected by the image pickup mode switch 12is determined. As a result of the determination, when the mode selectedby the image pickup mode switch 12 is the video image mode, the processproceeds to a step S103.

In the step S103, various set values of the video image mode are readfrom the internal nonvolatile memory 102 and are stored into the primarymemory 103. Then, the process proceeds to a step S104. The various setvalues of the video image mode include a field-angle set value V_(ang)and an image stabilization level. The field-angle set value V_(ang) ispreset to 90° in this embodiment. The image stabilization level isselected from among “Strong”, “Middle”, and “OFF”. In the step S104, anoperation of the image pickup driver 41 for the video image mode isstarted. And then, the process exits from this subroutine.

As a result of the determination in the step S102, when the modeselected by the image pickup mode switch 12 is the still image mode, theprocess proceeds to a step S106. In the step S106, various set values ofthe still image mode are read from the internal nonvolatile memory 102and are stored into the primary memory 103. Then, the process proceedsto a step S107. The various set values of the still image mode includethe field-angle set value V_(ang) and the image stabilization level. Thefield-angle set value V_(ang) is preset to 45° in this embodiment. Theimage stabilization level is selected from among “Strong”, “Middle”, and“OFF”. In the step S107, an operation of the image pickup driver 41 forthe still image mode is started. And then, the process exits from thissubroutine.

As the result of the determination in the step S102, when the modeselected by the image pickup mode switch 12 is the preset mode, theprocess proceeds to a step S108. The preset mode is one of the threeimage pickup modes that can be changed by the image pickup mode switch12. In the preset mode, the image pickup mode of the camera body 1 canbe changed by an external device like the display apparatus 800. Thatis, the preset mode is a mode for a custom image pickup operation. Sincethe camera body 1 is a compact wearable device, operation switches, asetting screen, etc. for changing advanced set values are not mounted onthe camera body 1. The advanced set values are changed by an externaldevice like the display apparatus 800.

For example, a case where the user would like to pick up a video imageat the field angle 90° and the field angle 110° continuously isconsidered. In such a case, the following operations are needed. Sincethe field angle is set to 90° in a regular video image mode, the userfirst performs the video image pickup operation in the regular videoimage mode, once finishes the video image pickup operation, displays thesetting screen on the display apparatus 800, and changes the field angleto 110° on the setting screen. However, the operations to the displayapparatus 800 during a certain event are troublesome.

In the meantime, when the preset mode is preset to a video image pickupoperation at the field angle 110°, the user can change the field anglein the video image pickup operation to 110° immediately by only slidingthe image pickup mode switch 12 to “Pre” after finishing the video imagepickup operation at the field angle 90°. That is, the user is notrequired to suspend the current operation and to perform theabove-mentioned troublesome operations.

It should be noted that contents of the preset mode may include theimage stabilization level, which is selected from among “Strong”,“Middle”, and “OFF”, and a set value of voice recognition that is notdescribed in this embodiment in addition to the field angle.

In the step S108, various set values of the preset mode are read fromthe internal nonvolatile memory 102 and are stored into the primarymemory 103. Then, the process proceeds to a step S109. The various setvalues of the preset mode include the field-angle set value V_(ang) andthe image stabilization level that is selected from among “Strong”,“Middle”, and “OFF”.

In the step S109, an operation of the image pickup driver 41 for thepreset mode is started. And then, the process exits from thissubroutine.

Hereinafter, the various set values of the video image mode read in thestep S103 will be described using FIG. 13 . FIG. 13 is a view showing amenu screen for setting the various set values of the video image modethat is displayed on the display unit 803 of the display apparatus 800before an image pickup operation of the camera body 1. The componentsthat have been described using FIG. 1D are indicated by the samereference numerals and their descriptions will be omitted. The displayunit 803 has a touch panel function and will be described under thepresumption that it functions by touch operations, such as a swipeoperation.

As shown in FIG. 13 , the menu screen includes a preview screen 831, azoom lever 832, a recording start/stop button 833, a switch 834, abattery residue indicator 835, a button 836, a lever 837, and an icondisplay area 838. The user can check the image picked up by the camerabody 1, a zoom amount, and a field angle on the preview screen 831.

The user can change a zoom setting (a field angle) by shifting the zoomlever 832 rightward or leftward. This embodiment describes a case wherethe field-angle set value V_(ang) can be selected from among 45°, 90°,110°, and 130°. In the meantime, the field-angle set value V_(ang) maybe set to a value other than the four values by operating the zoom lever832.

The recording start/stop button 833 is a toggle switch that has both ofthe functions of the start switch 14 and the stop switch 15. The switch834 is used to switch “OFF” and “ON” of the image stabilization process.The battery level indicator 835 displays battery level of the camerabody 1. The button 836 is used to change a mode.

The lever 837 is used to set the image stabilization level. Although theimage stabilization level can be set to “Strong” or “Middle” in thisembodiment, another image stabilization level, for example “Weak”, maybe set. Moreover, the image stabilization level may be set steplessly. Aplurality of thumbnail icons for preview are displayed in the icondisplay area 838.

FIG. 7C is a flowchart showing a subroutine of the face directiondetection process in the step S200 in FIG. 7A. Before describing thedetails of this process, a face direction detection method usinginfrared light will be described using FIG. 8A through FIG. 8K.

FIG. 8A is a view showing a visible light image of a user's face lookedat from the position of the face direction detection window 13. Theimage in FIG. 8A is identical to an image picked up by a visible-lightimage sensor on the assumption that the face direction detection window13 permits transmission of visible light and that the visible-lightimage sensor is mounted as a sensor of the infrared detection device 27.

The image in FIG. 8A includes a neck front part 201 above clavicles ofthe user, a root 202 of a jaw, a chin 203, and a face 204 including anose. FIG. 8B is a view showing a case where fluorescent lamps 205 in aroom appear as background in the visible-light image of the user shownin FIG. 8A.

The fluorescent lamps 205 around the user appear in the visible-lightimage in FIG. 8B. In this way, since various backgrounds appear in auser's image according to a use condition, it becomes difficult that theface direction detection unit 20 or the overall control CPU 101 cuts outa face image from a visible-light image. In the meantime, although thereis a technique that cuts such an image by using an AI etc., thetechnique is not suitable for the camera body 1 as a portable devicebecause the overall control CPU 101 is required to have highperformance.

Accordingly, the camera body 1 of the first embodiment detects a user'sface using an infrared image. Since the face direction detection window13 is constituted by a visible light cut filter, visible light is nottransmitted mostly. Accordingly, an image obtained by the infrareddetection device 27 is different from the images in FIG. 8A and FIG. 8B.

FIG. 8C is a view showing an infrared image obtained by imaging the userand the fluorescent lamps as the background shown in FIG. 8B onto thesensor of the infrared detection device 27 through the face directiondetection window 13 in a state where the infrared LEDs 22 are notlightened.

In the infrared image in FIG. 8C, the user's neck and jaw are dark. Inthe meantime, since the fluorescent lamps 205 emit an infrared componentin addition to the visible light, they are slightly bright.

FIG. 8D is a view showing an image obtained by imaging the user and thefluorescent lamps as the background shown in FIG. 8B onto the sensor ofthe infrared detection device 27 through the face direction detectionwindow 13 in a state where the infrared LEDs 22 are lightened.

In the image in FIG. 8D, the user's neck and jaw are bright. In themeantime, unlike FIG. 8C, the brightness around the fluorescent lamps205 has not changed.

FIG. 8E is a view showing a difference image that is calculated bysubtracting the image in FIG. 8C from the image in FIG. 8D. The user'sface emerges.

In this way, the overall control CPU 101 obtains the difference image(hereinafter also referred to as a face image) by calculating thedifference between the image formed on the sensor of the infrareddetection device 27 in the state where the infrared LEDs 22 arelightened and the image formed on the sensor in the state where theinfrared LEDs 22 are not lightened.

The face direction detection unit 20 of this embodiment employs a methodthat obtains a face image by extracting infrared reflection intensity asa two-dimensional image by the infrared detection device 27. The sensorof the infrared detection device 27 employs a configuration similar to ageneral image sensor and obtains a face image frame-by-frame. A verticalsynchronization signal (hereinafter referred to as a V-signal) thatobtains frame synchronization is generated by the infrared detectiondevice 27 and is output to the overall control CPU 101.

FIG. 9 is a timing chart showing timings of lighting and extinction ofthe infrared LEDs 22 and related signals.

A V-signal output from the infrared detection device 27, an H-positionof the image signal output from the sensor of the infrared detectiondevice 27, an IR-ON signal output to the infrared LED lighting circuit21 from the overall control CPU 101, and pickup image data output to theoverall control CPU 101 from the sensor of the infrared detection device27 are shown in FIG. 9 in the order from the top. The horizontal timeaxes of these four signals are identical. When the V-signal becomesHigh, timings of the frame synchronization and timings of lighting andextinction of the infrared LEDs 22 are obtained.

FIG. 9 shows a first face image obtainment period t1 and a second faceimage obtainment period t2.

The infrared detection device 27 controls the operation of the sensor sothat the H-position of the image signal will synchronize with theV-signal as shown in FIG. 9 . Since the sensor of the infrared detectiondevice 27 employs the configuration similar to a general image sensor asmentioned above and its operation is well-known, a detailed descriptionof the control method is omitted.

The overall control CPU 101 controls switching of the IR-ON signalbetween High and Low in synchronization with the V-signal. Specifically,the overall control CPU 101 outputs the IR-ON signal of Low to theinfrared LED lighting circuit 21 during the period t1 and outputs theIR-ON signal of High to the infrared LED lighting circuit 21 during thesecond period t2.

The infrared LED lighting circuit 21 lightens the infrared LEDs 22 toproject the infrared light 23 to the user during the High period of theIR-ON signal. In the meantime, the infrared LED lighting circuit 21extinguishes the infrared LEDs 22 during the Low period of the IR-ONsignal.

A vertical axis of the pickup image data indicates a signal intensitythat is a light receiving amount of the reflected light 25. Since theinfrared LEDs 22 are extinguished during the first period t1, noreflected light comes from the user's face and pickup image data asshown in FIG. 8C is obtained. In the meantime, since the infrared LEDs22 are lightened during the second period t2, the reflected light 25comes from the user's face and pickup image data as shown in FIG. 8D isobtained. Accordingly, the signal intensity in the period t2 increasesfrom the signal intensity in the period t1 by the reflected light 25from the user's face.

A face image indicated in the bottom in FIG. 9 is obtained bysubtracting the image pickup data during the first period t1 from theimage pickup data during the second period t2. As a result of thesubtraction, face image data in which only the component of thereflected light 25 from the user's face is extracted is obtained.

FIG. 7C shows the face direction detection process in the step S200 thatincludes the operations described using FIG. 8C through FIG. 8E and FIG.9 .

In a step S201, a timing V1 at which the first period t1 starts isobtained when the V-signal output from the infrared detection device 27becomes High. When the timing V1 is obtained, the process proceeds to astep S202.

In a step S202, the IR-ON signal is set to Low and is output to theinfrared LED lighting circuit 21. Thereby, the infrared LEDs 22 areextinguished.

In a step S203, one frame of pickup image data output from the infrareddetection device 27 during the first period t1 is read. The image datais temporarily stored into the primary memory 103 as Frame1.

In a step S204, a timing V2 at which the second period t2 starts isobtained when the V-signal output from the infrared detection device 27becomes High. When the timing V2 is obtained, the process proceeds to astep S205.

In the step S205, the IR-ON signal is set to High and is output to theinfrared LED lighting circuit 21. Thereby, the infrared LEDs 22 arelightened.

In a step S206, one frame of pickup image data output from the infrareddetection device 27 during the second period t2 is read. The image datais temporarily stored into the primary memory 103 as Frame2.

In a step S207, the IR-ON signal is set to Low and is output to theinfrared LED lighting circuit 21. Thereby, the infrared LEDs 22 areextinguished.

In a step S208, Frame1 and Frame2 are read from the primary memory 103,and light intensity Fn of the reflected light 25 from the usercorresponding to the face image shown in FIG. 9 is calculated bysubtracting Frame1 from Frame2. This process is generally called blacksubtraction.

In a step S209, a throat position (a neck rotation center) is extractedfrom the light intensity Fn. First, the overall control CPU 101 dividesthe face image into a plurality of distance areas that will be describedusing FIG. 8F on the basis of the light intensity Fn.

FIG. 8F is a view showing a result obtained by adjusting shades of thedifference image shown in FIG. 8E so as to fit with a scale of lightintensity of the reflected light 25 of the infrared light 23 projectedto the face and neck of the user. FIG. 8F shows light intensitydistribution about sections of the face and neck of the user.

The face image on the left side in FIG. 8F shows the light intensitydistribution of the reflected light 25 in the face image shown in FIG.8E by gray steps applied to the respective divided areas. An Xf axis isoriented in a direction from the central part of the user's neck towardthe chin.

In a graph on the right side in FIG. 8F, a horizontal axis shows thelight intensity on the Xf axis of the face image and a vertical axisshows the Xf axis. The light intensity shown by the horizontal axisincreases as going rightward.

The face image in FIG. 8F is divided into six areas (distance areas) 211through 216 according to the light intensity. The area 211 is an areawhere the light intensity is the strongest and is shown by white amongthe gray steps. The area 212 is an area where the light intensity fallsslightly than the area 211 and is shown by quite bright gray among thegray steps. The area 213 is an area where the light intensity fallsstill more than the area 212 and is shown by bright gray among the graysteps. The area 214 is an area where the light intensity falls stillmore than the area 213 and is shown by middle gray among the gray steps.The area 215 is an area where the light intensity falls still more thanthe area 214 and is shown by slightly dark gray among the gray steps.The area 216 is an area where the light intensity is the weakest and isshown by the darkest gray among the gray steps. The area above the area216 is shown by black showing no light intensity.

The light intensity will be described in detail using FIG. 10A throughFIG. 10D. FIG. 10A through FIG. 10D are views describing movement of theuser's face in the vertical direction and show states observed from theleft side of the user.

FIG. 10A is a view showing a state where the user faces the front. Thereis the image-pickup/detection unit 10 in front of the clavicles of theuser. Moreover, the infrared light 23 of the infrared LEDs 22 irradiatesthe lower part of the user's head from the face direction detectionwindow 13 mounted in the upper portion of the image-pickup/detectionunit 10. A distance Dn from the face direction detection window 13 tothe throat 200 above the clavicles of the user, a distance Db from theface direction detection window 13 to the root 202 of the jaw, and adistance Dc from the face direction detection window 13 to the chin 203satisfy a relation of Dn<Db<Dc. Since light intensity is in inverseproportion to the square of distance, the light intensity in the imageformed by the reflected light 25 from the infrared irradiation surface24 on the sensor becomes gradually weaker in the order of the throat200, the root 202 of the jaw, and the chin 203. Moreover, since thedistance from the face direction detection window 13 to the face 204including the nose is still longer than the distance Dc, the lightintensity in the image corresponding to the face 204 becomes stillweaker. That is, in the case as shown in FIG. 10A, the image having thelight intensity distribution shown in FIG. 8F is obtained.

It should be noted that the configuration of the face directiondetection unit 20 is not limited to the configuration shown in thisembodiment as long as the face direction of the user can be detected.For example, an infrared pattern may be projected from the infrared LEDs22, and the sensor of the infrared detection device 27 may detect theinfrared pattern reflected from an irradiation target. In this case, itis preferable that the sensor of the infrared detection device 27 isconstituted by a structural optical sensor. Moreover, the sensor of theinfrared detection device 27 may be a sensor that compares the phase ofthe infrared light 23 and the phase of the reflected light 25. Forexample, a ToF (Time of Flight) sensor may be employed.

Next, the extraction of the throat position in the step S209 in FIG. 7Cwill be described using FIG. 8G. A left image in FIG. 8G is obtained bysuperimposing the reference numerals denoting the parts of the user'sbody shown in FIG. 10A, a double circle showing the throat position, anda black circle showing the chin position on FIG. 8F.

The white area 211 corresponds to the throat 200 (FIG. 10A), the quitebright gray area 212 corresponds to the neck front part 201 (FIG. 10A),and the bright gray area 213 corresponds to the root 202 of the jaw(FIG. 10A). Moreover, the middle gray area 214 corresponds to the chin203 (FIG. 10A), and the slightly dark gray area 215 corresponds to a liplocated in the lower part of the face 204 (FIG. 10A) and a face lowerpart around the lip. Furthermore, the darkest gray area 216 correspondsto the nose located in the center of the face 204 (FIG. 10A) and a faceupper part around the nose.

Since the difference between the distances Db and Dc is relatively smallas compared with the differences between the other distances from theface direction detection window 13 to other parts of the user as shownin FIG. 10A, the difference between the reflected light intensities inthe bright gray area 213 and the middle gray area 214 is also small.

In the meantime, since the distance Dn is the shortest distance amongthe distances from the face direction detection window 13 to the partsof the user as shown in FIG. 10A, the reflection light intensity in thewhite area 211 corresponding to the throat 200 becomes the strongest.

Accordingly, the overall control CPU 101 determines that the area 211corresponds to the throat 200 and its periphery. And then, the overallcontrol CPU 101 sets a throat position 206 indicated by the doublecircle in FIG. 8G, which is located at the center in the lateraldirection in the area 211 and is the nearest to theimage-pickup/detection unit 10, as the position of the neck rotationcenter. The processes up to the moment are the contents performed in thestep S209 in FIG. 7C.

Next, the extraction of the chin position in the step S210 in FIG. 7Cwill be described using FIG. 8G. In the image in FIG. 8G, the middlegray area 214 that is brighter than the area 215 corresponding to theface lower part including the lip of the face 204 includes the chin. Agraph on the right side in FIG. 8G shows that the light intensity fallssharply in the area 215 adjacent to the area 214 because the change rateof the distance from the face direction detection window 13 becomeslarge.

The overall control CPU 101 determines that the brighter area 214adjacent to the area 215 in which the light intensity falls sharply is achin area. Furthermore, the overall control CPU 101 calculates(extracts) the position (indicated by the black circle shown in FIG.8G), which is located at the center in the lateral direction in the area214 and is the farthest from the throat position 206, as a chin position207.

For example, FIG. 8H and FIG. 8I show changes in directing the face tothe right. FIG. 8H is a view showing a difference image calculated bythe similar method as FIG. 8E in directing the user's face to the right.FIG. 8I is a view showing a result obtained by adjusting shades of thedifference image in FIG. 8H so as to fit with a scale of lightintensities of reflected components of the infrared light projected tothe face and neck of the user and by superimposing the double circleshowing the throat position 206 as the position of the neck rotationcenter and the black circle showing a chin position 207 r.

Since the user's face is directed to the right, the area 214 moves to anarea 214 r shown in FIG. 8I that is located in the left side when it islooked up from the image-pickup/detection unit 10. The area 215corresponding to the face lower part including the lip in the face 204also moves to an area 215 r that is located in the left side when it islooked up from the image-pickup/detection unit 10.

Accordingly, the overall control CPU 101 determines that the brighterarea 214 r adjacent to the area 215 r in which the light intensity fallssharply is the chin area. Furthermore, the overall control CPU 101calculates (extracts) the position (indicated by the black circle shownin FIG. 8I), which is located at the center in the lateral direction inthe area 214 r and is the farthest from the throat position 206, as thechin position 207 r.

After that, the overall control CPU 101 finds a moving angle θr thatshows the rotational movement to the right from the chin position 207 inthe image in FIG. 8G to the chin position 207 r in FIG. 8I around thethroat position 206. As shown in FIG. 8I, the moving angle θr is anangle of movement of the user's face in the lateral direction.

According to the above-mentioned method, the angle of face (hereinafter,referred to as a face angle) of the user in the lateral direction iscalculated in the step S210 from the chin position detected by theinfrared detection device 27 of the face direction detection unit (athree-dimensional detection sensor) 20.

Next, detection of the face directed upward will be described. FIG. 10Bis a view showing a state where the user directs the face horizontally.FIG. 10C is a view showing a state where the user directs the faceupward by 33° from the horizontal direction.

The distance from the face direction detection window 13 to the chin 203is Ffh in FIG. 10B, and the distance from the face direction detectionwindow 13 to a chin 203 u is Ffu in FIG. 10C. Since the chin 203 u movesupwardly together with the face, the distance Ffu becomes longer thanthe distance Ffh as shown in FIG. 10C.

FIG. 8J is a view showing an image of the user who directs the faceupward by 33° from the horizontal direction viewed from the facedirection detection window 13. Since the user directs the face upward asshown in FIG. 10C, the face 204 including the lip and nose cannot beseen from the face direction detection window 13 located under theuser's jaw. The chin 203 and its neck side are seen. FIG. 8K showsdistribution of the light intensity of the reflected light 25 inirradiating the user in the state shown in FIG. 10C with the infraredlight 23. An image on the left side in FIG. 8K is a view showing aresult obtained by adjusting shades of the difference image calculatedby the same method as FIG. 8E so as to fit with a scale of lightintensities of reflected components of the infrared light projected tothe face and neck of the user and by superimposing the double circleshowing the throat position 206 and the black circle showing a chinposition 207 u. Two graphs in FIG. 8K show density changes of the leftimage. The left graph is equivalent to the graph in FIG. 8F and theright graph is equivalent to the graph in FIG. 8G.

Six areas 211 u, 212 u, 213 u, 214 u, 215 u, and 216 u corresponding tothe light intensities in FIG. 8K are indicated by adding “u” to thereference numerals of the same light intensity areas shown in FIG. 8F.Although the light intensity of the user's chin 203 is included in themiddle gray area 214 in FIG. 8F, it shifts to the black side and isincluded in the slightly dark gray area 215 u in FIG. 8K. In this way,since the distance Ffu is longer than the distance Ffh as shown in FIG.10C, the infrared detection device 27 can detect that the lightintensity of the reflected light 25 from the chin 203 is weakened ininverse proportion to the square of distance.

Next, detection of the face directed downward will be described. FIG.10D is a view showing a state that the user directs the face downward by22° from the horizontal direction. In FIG. 10D, a distance from the facedirection detection window 13 to a chin 203 d is Ffd.

Since the chin 203 d moves downwardly together with the face, thedistance Ffd becomes shorter than the distance Ffh as shown in FIG. 10Dand the light intensity of the reflected light 25 from the chin 203becomes stronger.

Returning back to FIG. 7C, in a step S211, the overall control CPU 101calculates the distance from the chin position to the face directiondetection window 13 on the basis of the light intensity of the chinposition detected by the infrared detection device 27 of the facedirection detection unit (three-dimensional detection sensor) 20. A faceangle in the vertical direction is also calculated on the basis of this.

In a step S212, the overall control CPU 101 stores the face angle in thelateral direction obtained in the step S210 and the face angle in thevertical direction obtained in the step S211 into the primary memory 103as a three-dimensional observation direction vi (“i” is arbitraryreference numeral) of the user. For example, when the user is observingthe front center, the face angle θh in the lateral direction is 0° andthe face angle θv in the vertical direction is 0°. Accordingly, anobservation direction vo in this case is represented by vectorinformation [0°, 0°]. Moreover, when the user is observing a right 45°direction, an observation direction Vr is represented by vectorinformation [45°, 0°].

Although the face angle in the vertical direction is calculated bydetecting the distance from the face direction detection window 13 inthe step S211, the face angle may be calculated by another method. Forexample, change of the face angle may be calculated by comparing changelevels of the light intensity of the chin 203. That is, the change ofthe face angle may be calculated by comparing a gradient CDh of thereflected light intensity from the root 202 of the jaw to the chin 203in the graph in FIG. 8G with a gradient CDu of the reflected lightintensity from the root 202 of the jaw to the chin 203 in the graph inFIG. 8K.

FIG. 7D is a flowchart showing a subroutine of therecording-direction/area determination process in the step S300 in FIG.7A. Before describing details of this process, a superwide-angle imagethat is subjected to determine a recording direction and a recordingarea in this embodiment will be described first using FIG. 11A.

In the camera body 1 of this embodiment, the image pickup unit 40 picksup a superwide-angle image of the periphery of theimage-pickup/detection unit 10 using the superwide-angle image pickuplens 16. An image of an observation direction can be obtained byextracting a part of the superwide-angle image.

FIG. 11A is a view showing a target visual field 125 set in asuperwide-angle image picked up by the image pickup unit 40 in a casewhere the user faces the front.

As shown in FIG. 11A, a pixel area 121 that can be picked up by thesolid state image sensor 42 is a rectangular area. Moreover, aneffective projection area (a predetermined area) 122 is an area of acircular half-celestial sphere image that is a fish-eye image projectedonto the solid state image sensor 42 by the image pickup lens 16. Theimage pickup lens 16 is adjusted so that the center of the pixel area121 will match the center of the effective projection area 122.

The outermost periphery of the circular effective projection area 122shows a position where an FOV (field of view) angle is 180°. When theuser is looking at the center in both the vertical and horizontaldirections, an angular range of the target visual field 125 that ispicked up and recorded becomes 90° (a half of the FOV angle) centered onthe center of the effective projection area 122. It should be noted thatthe image pickup lens 16 of this embodiment can also introduce lightoutside the effective projection area 122 and can project light withinthe maximum FOV angle 192° onto the solid state image sensor 42 as afish-eye image. However, the optical performance falls greatly in thearea outside the effective projection area 122. For example, resolutionfalls extremely, light amount falls, and distortion increases.Accordingly, in this embodiment, an image of an observation direction isextracted as a recording area only from the inside of the image(hereinafter referred to as a superwide-angle image, simply) projectedin the pixel area 121 within the half-celestial sphere image displayedon the effective projection area 122.

Since the size of the effective projection area 122 in the verticaldirection is larger than the size of the short side of the pixel area121, the upper and lower ends of the image in the effective projectionarea 122 are out of the pixel area 121 in this embodiment. However, therelationship between the areas is not limited to this. For example, theoptical system may be designed so that the entire effective projectionarea 122 will be included in the pixel area 121 by changing theconfiguration of the image pickup lens 16. Invalid pixel areas 123 areparts of the pixel area 121 that are not included in the effectiveprojection area 122.

The target visual field 125 shows an area of an image of a user'sobservation direction that will be extracted from the superwide-angleimage. The target visual field 125 is prescribed by left, right, upper,and lower field angles (45° in this case, the FOV angle 90°) centeringon the observation direction. In the example of FIG. 11A, since the userfaces the front, the center of the target visual field 125 becomes theobservation direction vo that matches the center of the effectiveprojection area 122.

The superwide-angle image shown in FIG. 11A includes an A-object 131that is a child, a B-object 132 that shows steps that the child who isthe A-object is trying to climb, and a C-object 133 that islocomotive-type playground equipment.

Next, the recording-direction/area determination process in the stepS300 in FIG. 7A that is executed to obtain an image of an observationdirection from the superwide-angle image described using FIG. 11A isshown in FIG. 7D. Hereinafter, this process is described using FIG. 12Athrough FIG. 12G that show concrete examples of the target visual field125.

In a step S301, a field-angle set value V_(ang) that is set in advanceis obtained by reading from the primary memory 103.

In this embodiment, the internal nonvolatile memory 102 stores all theavailable field angles (45°, 90°, 110°, and 130°) as field-angle setvalues V_(ang). The image extraction/development unit 50 extracts animage of an observation direction in an area defined by the field-angleset value V_(ang) from the superwide-angle image. Moreover, thefield-angle set value V_(ang) included in the various set values readfrom the internal nonvolatile memory 102 in one of the steps S103, S106,and S108 in FIG. 7B is established and is being stored in the primarymemory 103.

Moreover, in the step S301, the observation direction vi determined inthe step S212 is determined as the recording direction, an image in thetarget visual field 125 of which the center is designated by theobservation direction vi and of which an area is defined by the obtainedfield-angle set value V_(ang) is extracted from the superwide-angleimage, and the extracted image is stored into the primary memory 103.

For example, when the field-angle set value V_(ang) is 90° and theobservation direction vo (vector information [0°, 0°]) is detectedthrough the face direction detection process (FIG. 7C), the targetvisual field 125 of which the angular widths are 45° in left and rightand are 45° in up and down (FIG. 11A) is established centering on thecenter O of the effective projection area 122. FIG. 11B is a viewshowing the image in the target visual field 125 extracted from thesuperwide-angle image in FIG. 11A. That is, the overall control CPU 101sets the angle of the face direction detected by the face directiondetection unit 20 to the observation direction vi that is the vectorinformation showing the relative position of the target visual field 125with respect to the superwide-angle image.

In the case of the observation direction vo, since the influence of theoptical distortion caused by the image pickup lens 16 can be disregardedmostly, the shape of the established target visual field 125 is almostidentical to the shape of a target visual field 125 o (FIG. 12A) afterconverting the distortion in a step S303 mentions later. Hereinafter, atarget visual field after converting the distortion in the case of theobservation direction vi is called a target visual field 125 i.

Next, an image stabilization level that is set in advance is obtained byreading from the primary memory 103 in a step S302.

In this embodiment, as mentioned above, the image stabilization levelincluded in the various setting values read from the internalnonvolatile memory 102 in one of the steps S103, S106, and S108 isestablished and is being stored in the primary memory 103.

Moreover, in the step S302, an image-stabilization-margin pixel numberPis is set on the basis of the obtained image stabilization level.

In the image stabilization process, an image following in a directionopposite to a blur direction is obtained according to a blur amount ofthe image-pickup/detection unit 10. Accordingly, in this embodiment, animage stabilization margin required for the image stabilization isestablished around the target visual field 125 i.

Moreover, in this embodiment, a table that keeps values of theimage-stabilization-margin pixel number Pis in association withrespective image stabilization levels is stored in the internalnonvolatile memory 102. For example, when the image stabilization levelis “middle”, an image stabilization margin of which width is “100pixels” that is the image-stabilization-margin pixel number Pis readfrom the above-mentioned table is established around the target visualfield.

FIG. 12E is a view showing an example that gives an image stabilizationmargin corresponding to a predetermined image stabilization level aroundthe target visual field 125 o shown in FIG. 12A. Hereinto, a case wherethe image stabilization level is “middle”, i.e., where theimage-stabilization-margin pixel number Pis is “100 pixels” will bedescribed.

As shown by a dotted line in FIG. 12E, an image stabilization margin 126o of which the width is “100 pixels” that is theimage-stabilization-margin pixel number Pis is established at the left,right, upper, and lower sides of the target visual field 125 o.

FIG. 12A and FIG. 12E show the case where the observation direction vimatches the center O (the optical axis center of the image pickup lens16) of the effective projection area 122 for simplification of thedescription. In the meantime, when the observation direction vi isdirected to a periphery of the effective projection area 122, theconversion to reduce the influence of optical distortion is required.

In the step S303, the shape of the target visual field 125 establishedin the step S301 is corrected (converts distortion) in consideration ofthe observation direction vi and the optical property of the imagepickup lens 16 to generate the target visual field 125 i. Similarly, theimage-stabilization-margin pixel number Pis set in the step S302 is alsocorrected in consideration of the observation direction vi and theoptical property of the image pickup lens 16.

For example, the field-angle set value V_(ang) shall be 90° and the usershall observe a right 45° direction from the center o. In this case, theobservation direction Vr (vector information [45°, 0°]) is determined inthe step S212, and the area of 45° in left and right and 45° in up anddown centering on the observation direction Vr becomes the target visualfield 125. Furthermore, the target visual field 125 is corrected to thetarget visual field 125 r shown in FIG. 12B in consideration of theoptical property of the image pickup lens 16.

As shown in FIG. 12B, the target visual field 125 r becomes wider towardthe periphery of the effective projection area 122. And the position ofthe observation direction Vr approaches inside a little from the centerof the target visual field 125 r. This is because the optical design ofthe image pickup lens 16 in this embodiment is close to that of astereographic projection fish-eye lens. It should be noted that contentsof the correction depend on the optical design of the image pickup lens16. If the image pickup lens 16 is designed as an equidistant projectionfish-eye lens, an equal-solid-angle projection fish-eye lens, or anorthogonal projection fish-eye lens, the target visual field 125 iscorrected according to its optical property.

FIG. 12F is a view showing an example that gives an image stabilizationmargin 126 r corresponding to the same image stabilization level“middle” of the image stabilization margin in FIG. 12E around the targetvisual field 125 r shown in FIG. 12B.

The image stabilization margin 126 o (FIG. 12E) is established at theleft, right, upper, and lower sides of the target visual field 125 owith the width of “100 pixels” that is the image-stabilization-marginpixel number Pis. As compared with this, the image-stabilization-marginpixel number Pis of the image stabilization margin 126 r (FIG. 12F) iscorrected so as to increase toward the periphery of the effectiveprojection area 122.

In this way, the shape of the image stabilization margin establishedaround the target visual field 125 r is also corrected as with the shapeof the target visual field 125 r so that the correction amount willincrease toward the periphery of the effective projection area 122 asshown by the image stabilization margin 126 r in FIG. 12F. This is alsobecause the optical design of the image pickup lens 16 in thisembodiment is close to that of a stereographic projection fish-eye lens.It should be noted that contents of the correction depend on the opticaldesign of the image pickup lens 16. If the image pickup lens 16 isdesigned as an equidistant projection fish-eye lens, anequal-solid-angle projection fish-eye lens, or an orthogonal projectionfish-eye lens, the image stabilization margin 126 r is correctedaccording to its optical property.

The process executed in the step S303 that switches successively theshapes of the target visual field 125 and its image stabilization marginin consideration of the optical property of the image pickup lens 16 isa complicated process. Accordingly, in this embodiment, the process inthe step S303 is executed using a table that keeps shapes of the targetvisual field 125 i and its image stabilization margin for everyobservation direction vi stored in the internal nonvolatile memory 102.It should be noted that the overall control CPU 101 may have a computingequation corresponding to the optical design of the image pickup lens16. In such a case, the overall control CPU 101 can calculate an opticaldistortion value using the computing equation.

In a step S304, a position and size of an image recording frame arecalculated. As mentioned above, the image stabilization margin 126 i isestablished around the target visual field 125 i. However, when theposition of the observation direction vi is close to the periphery ofthe effective projection area 122, the shape of the image stabilizationmargin becomes considerably special as shown by the image stabilizationmargin 126 r, for example.

The overall control CPU 101 can extract an image only in such aspecial-shaped area and apply the development process to the extractedimage. However, it is not general to use an image that is notrectangular in recording as image data in the step S600 or intransmitting image data to the display apparatus 800 in the step S700.Accordingly, in the step S304, the position and size of the imagerecording frame 127 i of a rectangular shape that includes the entireimage stabilization margin 126 i are calculated.

FIG. 12F shows the image recording frame 127 r that is calculated in thestep S304 to the image stabilization margin 126 r by an alternate longand short dash line.

In a step S305, the position and size of the image recording frame 127 ithat are calculated in the step S304 are recorded into the primarymemory 103.

In this embodiment, an upper-left coordinate (Xi, Yi) of the imagerecording frame 127 i in the superwide-angle image is recorded as theposition of the image recording frame 127 i, and a lateral width WXi anda vertical width WYi that start from the coordinate (Xi, Yi) arerecorded as the size of the image recording frame 127 i. For example, acoordinate (Xr, Yr), a lateral width WXr, and a vertical width WYr ofthe image recording frame 127 r shown in FIG. 12F are recorded in thestep S305. It should be noted that the coordinate (Xi, Yi) is a XYcoordinate of which an origin is a predetermined reference point,specifically the optical center of the image pickup lens 16.

When the image stabilization margin 126 i and the image recording frame127 i have been determined in this way, the process exits from thissubroutine shown in FIG. 7D.

In the description so far, the observation directions of which thehorizontal angle is 0°, such as the observation direction v0 (the vectorinformation [0°, 0°]) and the observation direction Vr (the vectorinformation [45°, 0°]), have been described for simplifying thedescription of the complicated optical distortion conversion. In themeantime, an actual observation direction vi of the user is arbitrary.Accordingly, the recording area development process executed in a casewhere the horizontal angle is not 0° will be described hereinafter. Forexample, when the field-angle set value V_(ang) is 90° and theobservation direction vm is [−42°, −40°], the target visual field 125 mappears as shown in FIG. 12C.

Moreover, even when the observation direction vm (the vector information[−42°, −40°]) is the same as the target visual field 125 m, when thefield-angle set value V_(ang) is 45°, a target visual field 128 m, whichis slightly smaller than the target visual field 125 m, appears as shownin FIG. 12D. Furthermore, an image stabilization margin 129 m and animage recording frame 130 m are established around the target visualfield 128 m as shown in FIG. 12G.

Since the process in the step S400 is a fundamental image pickupoperation and employs a general sequence of the image pickup unit 40,its detailed description is omitted. It should be noted that the imagesignal processing circuit 43 in the image pickup unit 40 in thisembodiment also performs a process that converts signals of an inherentoutput format (standard examples: MIPI, SLVS) output from the solidstate image sensor 42 into pickup image data of a general sensor readingsystem.

When the video image mode is selected by the image pickup mode switch12, the image pickup unit 40 starts recording in response to a press ofthe start switch 14. After that, the recording is finished when the stopswitch 15 is pressed. In the meantime, when the still image mode isselected by the image pickup mode switch 12, the image pickup unit 40picks up a static image every time when the start switch 14 is pressed.

FIG. 7E is a flowchart showing a subroutine of the recording-areadevelopment process in the step S500 in FIG. 7A.

In a step S501, Raw data of the entire area of the pickup image data(superwide-angle image) generated by the image pickup unit 40 in thestep S400 is obtained and is input into an image capturing unit called ahead unit (not shown) of the overall control CPU 101.

Next, in a step S502, the image within the image recording frame 127 iis extracted from the superwide-angle image obtained in the step S501 onthe basis of the coordinate (Xi, Yi), lateral width WXi, and verticalwidth WYi that are recorded into the primary memory 103 in the stepS305. After the extraction, the processes in steps S503 through S508mentioned later are executed only to the pixels within the imagestabilization margin 126 i. In this way, a crop development process(FIG. 7F) consisting of the steps S502 through S508 begins.

This can reduce a calculation amount significantly as compared with acase where the development process is executed to the entire area of thesuperwide-angle image read in the step S501. Accordingly, calculationtime and electric power consumption can be reduced.

As shown in FIG. 7F, when the video image mode is selected by the imagepickup mode switch 12, the processes of the steps S200 and S300 and theprocess of the step S400 are executed in parallel by the same frame rateor different frame rates. Whenever the Raw data of the entire area ofone frame generated by the image pickup unit 40 is obtained, the cropdevelopment process is executed on the basis of the coordinate (Xi, Yi),lateral width WXi, and vertical width WYi that are recorded in theprimary memory 103 at that time point.

When the crop development process is started to the pixels within theimage stabilization margin 126 i, and when the part within the imagerecording frame 127 i is extracted in the step S502, color interpolationthat interpolates data of color pixels arranged in the Bayer arrangementis executed in the step S503. After that, a white balance is adjusted ina step S504, and then, a color conversion is executed in a step S505. Ina step S506, gamma correction that corrects gradation according to agamma correction value set up beforehand is performed. In a step S507,edge enhancement is performed in accordance with an image size.

In the step S508, the image data is converted into a data format thatcan be stored primarily by applying processes like compression. Theconverted image data is stored into the primary memory 103. After that,the process exits from the subroutine. Details of the data format thatcan be stored primarily will be mentioned later.

The order and presences of the processes in the crop development processexecuted in the steps S503 through S508 may be set up according to theproperty of the camera system and they do not restrict the presentinvention. Moreover, when the video image mode is selected, theprocesses of the steps S200 through S500 are repeatedly executed untilthe recording is finished.

According to this process, the calculation amount is significantlyreduced as compared with a case where the development process isexecuted to the entire area read in the step S501. Accordingly, aninexpensive and low-power consumption microcomputer can be employed asthe overall control CPU 101. Moreover, heat generation in the overallcontrol CPU 101 is reduced and the life of the battery 94 becomeslonger.

Moreover, in order to reduce a control load on the overall control CPU101, the optical correction process (the step S800 in FIG. 7A) and theimage stabilization process (the step S900 in FIG. 7A) to the image arenot executed by the camera body 1 in this embodiment. These processesare executed by the display-apparatus controller 801 after transferringthe image to the display apparatus 800. Accordingly, if only data of apartial image extracted from a projected superwide-angle image istransferred to the display apparatus 800, neither the optical correctionprocess nor the image stabilization process can be executed. That is,since the data of the extracted image does not include positioninformation that will be substituted to a formula of the opticalcorrection process and will be used to refer the correction table of theimage stabilization process, the display apparatus 800 cannot executethese processes correctly. Accordingly, in this embodiment, the camerabody 1 transmits correction data including information about anextraction position of an image from a superwide-angle image togetherwith data of the extracted image to the display apparatus 800.

When the extracted image is a still image, since the still image datacorresponds to the correction data one-to-one, the display apparatus 800can execute the optical correction process and image stabilizationprocess correctly, even if these data are separately transmitted to thedisplay apparatus 800. In the meantime, when the extracted image is avideo image, if the video image data and the correction data areseparately transmitted to the display apparatus 800, it becomesdifficult to determine correspondence between each frame of the videoimage data and the correction data. Particularly, when a clock rate ofthe overall control CPU 101 in the camera body 1 slightly differs from aclock rate of the display-apparatus controller 801 in the displayapparatus 800, the synchronization between the overall control CPU 101and the display-apparatus controller 801 will be lost during the videoimage pickup operation for several minutes. This may cause a defect thatthe display-apparatus controller 801 corrects a frame with correctiondata different from the corresponding correction data.

Accordingly, in this embodiment, when transmitting data of an extractedvideo image to the display apparatus 800, the camera body 1 gives itscorrection data appropriately to the data of the video image.Hereinafter, the method is described.

FIG. 14 is a flowchart showing the subroutine of the primary recordingprocess in the step S600 in FIG. 7A. Hereinafter, this process will bedescribed by also referring to FIG. 15 . FIG. 14 shows the process of acase where the video image mode is selected by the image pickup modeswitch 12. When the still image mode is selected, this process startsfrom a step S601 and is finished after a process of a step S606.

In a step S601 a, the overall control CPU 101 reads an image of oneframe to which the processes in steps S601 through S606 have not beenapplied from among the video image developed in the recording areadevelopment process (FIG. 7E). Moreover, the overall control CPU 101generates correction data that is metadata of the read frame.

In the step S601, the overall control CPU 101 attaches the informationabout the extraction position of the image of the frame read in the stepS601 a to the correction data. The information attached in this step isthe coordinate (Xi, Yi) of the image recording frame 127 i obtained inthe step S305. It should be noted that the information attached in thisstep may be the vector information that shows the observation directionvi.

In a step S602, the overall control CPU 101 obtains an opticalcorrection value. The optical correction value is the optical distortionvalue set up in the step S303. Alternatively, the optical correctionvalue may be a correction value corresponding to the lens opticalproperty, such as a marginal-light-amount correction value or adiffraction correction value.

In a step S603, the overall control CPU 101 attaches the opticalcorrection value used for the distortion conversion in the step S602 tothe correction data.

In a step S604, the overall control CPU 101 determines whether the imagestabilization mode is effective. Specifically, when the imagestabilization mode set up in advance is “Middle” or “Strong”, it isdetermined that the image stabilization mode is effective and theprocess proceeds to a step S605. In the meantime, when the imagestabilization mode set up in advance is “OFF”, it is determined that theimage stabilization mode is not effective and the process proceeds tothe step S606. The reason why the step S605 is skipped when the imagestabilization mode is “OFF” is because the calculation data amount ofthe overall control CPU 101 and the data amount of the wirelesscommunication are reduced and the power consumption and heat generationof the camera body 1 can be reduced by skipping the step S605. Althoughthe reduction of the data used for the image stabilization process isdescribed, the data about the marginal-light-amount value or the dataabout the diffraction correction value obtained as the opticalcorrection value in the step S602 may be reduced.

Although the image stabilization mode is set up by the user's operationto the display apparatus 800 in advance in this embodiment, it may beset up as a default setting of the camera body 1. Moreover, when thecamera system is configured to switch the effectiveness of the imagestabilization process after transferring image data to the displayapparatus 800, the process may directly proceed to the step S605 fromthe step S603 by omitting the step S604.

In the step S605, the overall control CPU 101 attaches the imagestabilization mode, which is obtained in the step S302, and the gyrodata, which is obtained during the pickup operation of the video imagein association with the frame that is read from the primary memory 813in the step S601 a, to the correction data.

In the step S606, the overall control CPU 101 updates a video file 1000(FIG. 15 ) by data obtained by encoding the image data of the frame readin the step S601 a and the correction data to which the various data areattached in the steps S601 through S605. It should be noted that when afirst frame of the video image is read in the step S601 a, the videofile 1000 is generated in the step S606.

In a step S607, the overall control CPU 101 determines whether all theframes of the video image developed by the recording area developmentprocess (FIG. 7E) have been read. When not all the frames have beenread, the process returns to the step S601 a. In the meantime, when allthe frames have been read, the process exits from this subroutine. Thegenerated video file 1000 is stored into the internal nonvolatile memory102. The video file may be stored into the large-capacity nonvolatilememory 51 too in addition to the primary memory 813 and the internalnonvolatile memory 102. Moreover, the transmission process (the stepS700 in FIG. 7A) that transfers the generated image file 1000 to thedisplay apparatus 800 immediately is executed. The image file 1000 maybe stored into the primary memory 813 after transferring it to thedisplay apparatus 800.

In this embodiment, the encoding means to combine the image data and thecorrection data into one file. At that time, the image data may becompressed or the data file that is combined by the image data andcorrection data may be compressed.

FIG. 15 is a view showing a data structure of the video file 1000. Thevideo file 1000 consists of a header part 1001 and a frame part 1002.The frame part 1002 consists of frame data sets each of which consistsof an image of each frame and corresponding frame metadata. That is, theframe part 1002 includes frame data sets of the number of the totalframes of the video image.

In this embodiment, the frame metadata is information obtained byencoding correction data to which an extraction position (in-imageposition information), an optical correction value, and gyro data areattached if needed. However, the frame metadata is not limited to this.An information amount of the frame metadata may be changed. For example,other information may be added to the frame metadata according to theimage pickup mode selected by the image pickup mode switch 12.Alternatively, a part of the information in the frame metadata may bedeleted.

An offset value to the frame data sets of each frame or a head addressof each frame is recorded in the header part 1001. Alternatively,metadata like the time and size corresponding to the video file 1000 maybe stored in the header part 1001.

In the primary recording process (FIG. 14 ), the video file 1000 istransferred to the display apparatus 800 in this way. The video file 100includes data sets each of which consists of a frame of the video imagedeveloped by the recording area development process (FIG. 7E) and itsmetadata. Accordingly, even when the clock rate of the overall controlCPU 101 in the camera body 1 slightly differs from the clock rate of thedisplay-apparatus controller 801 in the display apparatus 800, thedisplay-apparatus controller 801 appropriately applies the correctionprocess to the video image developed in the camera body 1.

Although the optical correction value is included in the frame metadatain this embodiment, the optical correction value may be given to theentire video image.

FIG. 16 is a flowchart showing the subroutine of the transmissionprocess to the display apparatus 800 in the step S700 in FIG. 7A. FIG.16 shows the process of a case where the video image mode is selected bythe image pickup mode switch 12. It should be noted that when the stillimage mode is selected, this process starts from a process in a stepS702.

In a step S701, it is determined whether the image pickup process (thestep S400) of the video image by the image pickup unit 40 is finished oris under recording. When the video image is recording (during the videoimage pickup operation), the recording area development process (thestep S500) for each frame and the update of the image file 1000 (thestep S606) in the primary recording process (the step S600) are executedsequentially. Since a power load of wireless transmission is large, ifthe wireless transmission is performed during the video image pickupoperation in parallel, the battery 94 is needed to have large batterycapacity or a new measure against heat generation is needed. Moreover,from a viewpoint of arithmetic capacity, if the wireless transmission isperformed during the video image pickup operation in parallel, anarithmetic load will become large, which needs to prepare ahigh-specification CPU as the overall control CPU 101, increasing thecost.

In view of these points, in this embodiment, the overall control CPU 101proceeds with the process to a step S702 after the video image pickupoperation is finished (YES in the step S701), and establishes thewireless connection with the display apparatus 800. In the meantime, ifthe camera system of the embodiment has a margin in the electric powersupplied from the battery 94 and a new measure against heat generationis unnecessary, the overall control CPU 101 may beforehand establish thewireless connection with the display apparatus 800 when the camera body1 is started or before starting the recording.

In the step S702, the overall control CPU 101 establishes the connectionwith the display apparatus 800 through the high-speed wirelesscommunication unit 72 in order to transfer the video file 1000 havingmuch data volume to the display apparatus 800. It should be noted thatthe small-power wireless communication unit 71 is used for transmissionof a low-resolution image for checking a field angle to the displayapparatus 800 and is used for exchange of various set values with thedisplay apparatus 800. In the meantime, the small-power wirelesscommunication unit 71 is not used for transfer of the video file 1000because a transmission period becomes long.

In a step S703, the overall control CPU 101 transfers the video file1000 to the display apparatus 800 through the high-speed wirelesscommunication unit 72. When the transmission is finished, the overallcontrol CPU 101 proceeds with the process to a step S704. In the stepS704, the overall control CPU 101 closes the connection with the displayapparatus 800 and exits from this subroutine.

The case where one image file includes the images of all the frames ofone video image has been described so far. In the meantime, if therecording period of the video image is longer than several minutes, thevideo image may be divided by a unit time into a plurality of imagefiles. When the video file has the data structure shown in FIG. 15 ,even if one video image is transferred to the display apparatus 800 as aplurality of image files, the display apparatus 800 can correct thevideo image without the timing gap with the correction data.

FIG. 17 is a flowchart showing a subroutine of the optical correctionprocess in the step S800 in FIG. 7A. Hereinafter, this process will bedescribed by also referring to FIG. 18A through FIG. 18E. As mentionedabove, this process is executed by the display-apparatus controller 801of the display apparatus 800.

In a step S801, the display-apparatus controller 801 first receives thevideo file 1000 from the camera body 1 transferred in the transmissionprocess (the step S700) to the display apparatus 800. After that, thedisplay-apparatus controller 801 obtains the optical correction valueextracted from the received video file 1000.

Next, in a step S802, the display-apparatus controller 801 obtains animage (an image of one frame obtained by the video image pickupoperation) from the video file 1000.

In a step S803, the display-apparatus controller 801 corrects opticalaberration of the image obtained in the step S802 with the opticalcorrection value obtained in the step S801, and stores the correctedimage into the primary memory 813. When the extraction from the imageobtained in the step S802 is performed in the optical correction, animage area (extraction-development area) that is narrower than thedevelopment area (target visual field 125 i) determined in the step S303is extracted and is subjected to the process.

FIG. 18A through FIG. 18F are views for describing a process of applyingdistortion correction in the step S803 in FIG. 17 .

FIG. 18A is a view showing a position of an object 1401 at which theuser looks with a naked eye in picking up an image. FIG. 18B is a viewshowing an image of the object 1401 formed on the solid state imagesensor 42.

FIG. 18C is a view showing a development area 1402 in the image in FIG.18B. The development area 1402 is the extraction-development areamentioned above.

FIG. 18D is a view showing an extraction-development image obtained byextracting the image of the development area 1402. FIG. 18E is a viewshowing an image obtained by correcting distortion in theextraction-development image shown in FIG. 18D. Since an extractionprocess is performed in correcting distortion of theextraction-development image, a field angle of the image shown in FIG.18E becomes still smaller than that of the extraction-development imageshown in FIG. 18D.

FIG. 19 is a flowchart showing a subroutine of the image stabilizationprocess in the step S900 in FIG. 7A. Hereinafter, this process will bedescribed by also referring to FIG. 18F. As mentioned above, thisprocess is executed by the display-apparatus controller 801 of thedisplay apparatus 800.

In a step S901, the display-apparatus controller 801 obtains gyro dataof a current frame, gyro data of a previous frame, and a blur amountV_(n-1) ^(Det), which is calculated in a below-mentioned step S902 forthe previous frame, from the frame metadata of the video file 1000.After that, a rough blur amount V_(n) ^(Pre) is calculated from thesepieces of information. It should be noted that a current frame in thisembodiment is a frame under processing and that a previous frame is animmediately preceding frame.

In a step S902, the display-apparatus controller 801 calculates a fineblur amount V_(n) ^(Det) from the video file. A blur amount is detectedby calculating a moving amount of a feature point in the image from aprevious frame to a current frame.

A feature point can be extracted by a known method. For example, amethod using a luminance information image that is generated byextracting only luminance information of an image of a frame may beemployed. This method subtracts an image that shifts the originalluminance information image by one or several pixels from the originalluminance information image. A pixel of which an absolute value ofdifference exceeds a threshold is extracted as a feature point.Moreover, an edge extracted by subtracting an image generated byapplying a high-pass filter to the above-mentioned luminance informationimage from the original luminance information image may be extracted asa feature point.

Differences are calculated multiple times while shifting the luminanceinformation images of the current frame and previous frame by one orseveral pixels. The moving amount is obtained by calculating a positionat which the difference at the pixel of the feature point diminishes.

Since a plurality of feature points are needed as mentioned later, it ispreferable to divide each of the images of the present frame andprevious frame into a plurality of blocks and to extract a feature pointfor each block. A block division depends on the number of pixels andaspect ratio of the image. In general, 12 blocks of 4*3 or 54 blocks of9*6 are preferable. When the number of blocks is too small, trapezoidaldistortion due to a tilt of the image pickup unit 40 of the camera body1 and rotational blur around the optical axis, etc. cannot be correctedcorrectly. In the meantime, when the number of blocks is too large, asize of one block becomes small, which shortens a distance betweenadjacent feature points, causing an error. In this way, the optimalnumber of blocks is selected depending on the pixel number, ease ofdetection of feature points, a field angle of an object, etc.

Since the calculation of the moving amount needs a plurality ofdifference calculations while shifting the luminance information imagesof the current frame and previous frame by one or several pixels, thecalculation amount increases. Since the moving amount is actuallycalculated on the basis of the rough blur amount V_(n) ^(Pre) anddeviation (the number of pixels) therefrom, the difference calculationsare performed only near the rough blur amount, which can significantlyreduce the calculation amount.

Next, in a step S903, the display-apparatus controller 801 performs theimage stabilization process using the fine blur amount V_(n) ^(Det)obtained in the step S902. And then, the process exits form thissubroutine.

It should be noted that Euclidean transformation and affinetransformation that enable rotation and parallel translation, andprojective transformation that enables keystone correction are known asthe method of the image stabilization process.

Although the Euclid transformation can correct movement in an X-axisdirection and a Y-axis direction and rotation, it cannot correct blurcaused by camera shake of the image pickup unit 40 of the camera body 1in a front-back direction or directions of pan and tilt. Accordingly, inthis embodiment, the image stabilization process is executed using theaffine transformation that enables correction of skew. The affinetransformation from a coordinate (x, y) of the feature point used ascriteria to a coordinate (x′, y′) is expressed by the following formula100.

$\begin{matrix}{\begin{pmatrix}x^{\prime} \\y^{\prime} \\1\end{pmatrix} = {\begin{pmatrix}a & b & c \\d & e & f \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}} & {{Formula}100}\end{matrix}$

Affine coefficients of a 3*3 matrix of the formula 100 are computable ifdeviations of at least three feature points are detected. However, whenthe detected feature points are mutually near or are aligned on astraight line, the image stabilization process becomes inaccurate inareas distant from the feature points or distant from the straight line.Accordingly, it is preferable to select the feature points to bedetected that are mutually distant and do not lie on a straight line.Accordingly, when a plurality of feature points are detected, mutuallynear feature points are excluded and remaining feature points arenormalized by a least square method.

FIG. 18F is a view showing an image obtained by applying the imagestabilization process in the step S903 to the distortion-corrected imageshown in FIG. 18E. Since the extraction process is performed inexecuting the image stabilization process, a field angle of the imageshown in FIG. 18F becomes smaller than that of the image shown in FIG.18E.

It is available to obtain a high quality image of which blur iscorrected by performing such an image stabilization process.

In the above, the series of operations executed by the camera body 1 anddisplay apparatus 800 that are included in the camera system of thisembodiment have been described.

When the user selects the video image mode by the image pickup modeswitch 12 after turning the power switch 11 ON and observes the frontwithout turning the face in the vertical and horizontal directions, theface direction defection unit 20 detects the observation direction vo(vector information [0°, 0°]) as shown in FIG. 12A. After that, therecording-direction/field-angle determination unit 30 extracts the image(FIG. 11B) in the target visual field 125 o shown in FIG. 12A from thesuperwide-angle image projected onto the solid state image sensor 42.

After that, when the user starts observing the child (A-object 131) inFIG. 11A, for example, without operating the camera body 1, the facedirection detection unit 20 detects the observation direction vm (vectorinformation [−42°, −40°]) as shown in FIG. 11C. After that, therecording-direction/field-angle determination unit 30 extracts the image(FIG. 11C) in the target visual field 125 m from the superwide-angleimage picked up by the image pickup unit 40.

In this way, the display apparatus 800 applies the optical correctionprocess and image stabilization process to the extracted image of theshape depending on the observation direction in the steps S800 and S900.Thereby, even if the specification of the overall control CPU 101 of thecamera body 1 is low, the significantly distorted image in the targetvisual field 125 m (FIG. 11C) is converted into the image around thechild (A-object) 131 of which the blur and distortion are corrected asshown in FIG. 11D. That is, the user is able to obtain an image pickedup in the own observation direction, even if the user does not touch thecamera body 1 except to turn the power switch 11 ON and to select themode with the image pickup mode switch 12.

Hereinafter, the preset mode will be described. Since the camera body 1is a compact wearable device as mentioned above, operation switches, asetting screen, etc. for changing advanced set values are not mounted onthe camera body 1. Accordingly, in this embodiment, the advanced setvalues of the camera body 1 are changed using the setting screen (FIG.13 ) of the display apparatus 800 as an external device.

For example, a case where the user would like to change the field anglefrom 90° to 45° while picking up a video image continuously isconsidered. In such a case, the following operations are needed. Sincethe field angle is set to 90° in a regular video image mode, the userperforms the video image pickup operation in the regular video imagemode, once finishes the video image pickup operation, displays thesetting screen of the camera body 1 on the display apparatus 800, andchanges the field angle to 45° on the setting screen. However, thisoperation to the display apparatus 800 during the continuous imagepickup operation is troublesome and an image that the user wants to pickup may be missed.

In the meantime, when the preset mode is preset to a video image pickupoperation at the field angle of 45°, the user can change to a zoom-upvideo image pickup operation at the field angle of 45° immediately byonly sliding the image pickup mode switch 12 to “Pre” after finishingthe video image pickup operation at the field angle of 90°. That is, theuser is not required to suspend the current image pickup operation andto perform the above-mentioned troublesome operations.

The contents of the preset mode may include the image stabilizationlevel (“Strong”, “Middle”, or “OFF”) and a set value of voicerecognition that is not described in this embodiment in addition to thefield angle.

For example, when the user switches the image pickup mode switch 12 fromthe video image mode to the preset mode while continuously observing thechild (A-object) 131 in the previous image pickup situation, thefield-angle set value V_(ang) is changed from 90° to 45°. In this case,the recording-direction/field-angle determination unit 30 extracts theimage in the target visual field 128 m shown by a dotted line frame inFIG. 11E from the superwide-angle image picked up by the image pickupunit 40.

Also in the preset mode, the optical correction process and imagestabilization process are performed in the display apparatus 800 in thesteps S800 and S900. Thereby, even if the specification of the overallcontrol CPU 101 of the camera body 1 is low, the zoom-up image aroundthe child (A-object) 131 of which the blur and distortion are correctedas shown in FIG. 11F is obtained. Although the case where thefield-angle set value V_(ang) is changed from 90° to 45° in the videoimage mode has been described, the process in the still image mode issimilar. Moreover, a case where the field-angle set value V_(ang) of avideo image is 90° and the field-angle set value V_(ang) of a staticimage is 45° is also similar.

In this way, the user is able to obtain the zoom-up image that picks upthe own observation direction by just switching the mode with the imagepickup mode switch 12 of the camera body 1.

Although the case where the face direction detection unit 20 and theimage pickup unit 40 are integrally constituted in the camera body 1 isdescribed in this embodiment, the configuration is not limited to thisas long as the face direction detection unit 20 is worn on the user'sbody other than the head and the image pickup unit 40 is worn on theuser's body. For example, the image-pickup/detection unit 10 of thisembodiment can be worn on a shoulder or an abdomen. However, when theimage pickup unit 40 is worn on a right shoulder, an object of the leftside is obstructed by the head. In such a case, it is preferable that aplurality of image pickup units are worn on places including a rightshoulder. Moreover, when the image pickup unit 40 is worn on an abdomen,spatial parallax occurs between the image pickup unit 40 and the head.In such a case, it is preferable to perform a correction calculation ofthe observation direction that compensate such parallax as described ina third embodiment.

Hereinafter, a second embodiment will be described. The secondembodiment is basically identical to the first embodiment in theconfiguration, but an image-pickup/detection unit 1100 is provided inthe camera body 1 instead of the image-pickup/detection unit 10.Hereinafter, a component that has the same function and configuration asthe component already described in the first embodiment is indicated bythe same reference numeral, its description in the specification isomitted, and only a configuration peculiar to the second embodiment isdescribed.

FIG. 20 is an exploded perspective view schematically showing theconfiguration of the image-pickup/detection unit 1100 according to thesecond embodiment. As shown in FIG. 20 , the image-pickup/detection unit1100 has an image pickup unit for picking up an object and anobservation-direction detection unit for detecting an observationdirection of a user. The image pickup unit is provided with an imagepickup lens 1101 and an image sensor 1103 a. The observation-directiondetection unit is provided with a face direction detection lens 1102 andan image sensor (not shown). The image pickup lens 1101 and the imagesensor 1103 a are supported by a substrate 1103. Moreover, the facedirection detection lens 1102 and the image sensor (not shown) aresupported by a flexible substrate 1110. Furthermore, theimage-pickup/detection unit 1100 has a front cover 1104, a back cover1105, an upper cover 1109, and a lens barrier (protection member) 1111that protects the image pickup lens 1101.

Infrared LEDs (light emission members) 1110 a and 1110 b, whichirradiate each part of a user's face with infrared light, and a circuitfor lighting them are mounted on the flexible substrate 1110. It shouldbe noted that the number of the infrared LEDs is not restricted to two.The upper cover 1109 is a transparent plate member and is arranged abovethe face direction detection lens 1102 and the infrared LEDs 1110 a and1110 b. An opening 1104 a is formed at a place that faces the imagepickup lens 1101 in the front cover 1104. In the second embodiment, theimage pickup lens 1101 is a fish-eye lens and its part protrudes fromthe front cover 1104 to the outside through the opening 1104 a.

FIG. 21 is a view showing a state where a user 1112 wears the camerabody 1 equipped with the image-pickup/detection unit 1100. In FIG. 21 ,the components, such as the connection members 80L and 80R and thebattery unit 90 s, of the camera body 1 other thanimage-pickup/detection unit 1100 are not shown.

As shown in FIG. 21 , the user 1112 wears the camera body 1 so that theimage-pickup/detection unit 1100 will be located on a median plane(shown by an alternate long and short dash line in the drawing) of theuser 1112 and under a jaw. When the camera body 1 is viewed from thefront of the user 1112, an optical axis OA2 of the face directiondetection lens 1102 in the image-pickup/detection unit 1100approximately matches the median plane of the user 1112, and the facedirection detection lens 1102 points to the jaw of the user 1112.Moreover, an optical axis OA1 of the image pickup lens 1101 alsoapproximately matches the median plane of the user 1112. It should benoted that the image pickup lens 1101 is located under the facedirection detection lens 1102 when the user 1112 is equipped with thecamera body 1.

FIG. 22A and FIG. 22B are views for describing an operation of the lensbarrier 1111. FIG. 22A shows a non-image-pickup state, and FIG. 22Bshows an image-pickup state. Both show a case where theimage-pickup/detection unit 1100 is viewed from the front. Moreover,FIG. 23A is a sectional view along the optical axis OA1 and optical axisOA2 showing an internal configuration of the image-pickup/detection unit1100 in the non-image-pickup state. FIG. 23B is a sectional view alongthe optical axis OA1 and optical axis OA2 showing the internalconfiguration of the image-pickup/detection unit 1100 in theimage-pickup state. FIG. 24 is a front view showing theimage-pickup/detection unit 1100 when the front cover 1104 is removed.

The lens barrier 1111 is attached to the front cover 1104 so that it canbe slid in the vertical direction (a Y-direction in the drawing) whenthe user 1112 is equipped with the camera body 1. In thenon-image-pickup state (a first state), the lens barrier 1111 movesdownwardly and covers the image pickup lens 1101 of which the partprotrudes from the opening 1104 a (FIG. 22A, FIG. 23A). Moreover, in theimage-pickup state (a second state), the lens barrier 1111 movesupwardly to expose the image pickup lens 1101 to the outside, and a partof the lens barrier 1111 protrudes from the upper end of the front cover1104 (FIG. 22B, FIG. 23B). That is, the lens barrier 1111 varies betweenthe non-image-pickup state and image-pickup state by sliding in thevertical direction.

As shown in FIG. 23A and FIG. 23B, the face direction detection lens1102, the flexible substrate 1110, and the substrate 1103 are insertedand stored between the front cover 1104 and back cover 1105. Moreover,the upper cover 1109 covers the face direction detection lens 1102. Theoptical axis OA1 of the image pickup lens 1101 approximately matches animage pickup center of the image sensor 1103 a mounted on the substrate1103. Moreover, the face direction detection lens 1102 is arranged at aposition (upper position) offset in the Y-direction with respect to theimage pickup lens 1101. As mentioned above, the optical axis OA2 (shownby a broken line in the drawing) approximately matches the median planeof the user 1112 and the optical axis OA2 (shown by a broken line in thedrawing) also approximately matches the median plane of the user 112.Accordingly, the optical axis OA1 and optical axis OA2 intersect on themedian plane.

The face direction detection lens 1102 is supported by the flexiblesubstrate 1110 that implements the infrared LEDs 1110 a and 1110 bthrough the image sensor (not shown), and the flexible substrate 1110 isconnected to a connector 1103 b of the substrate 1103. Moreover, thememory media 1103 c is mounted on the substrate 1103.

As with the first embodiment, also in the second embodiment, the facedirection detection lens 1102 is located just under the jaw of the user1112 and points to the jaw of the user 1112 when the user 1112 wears thecamera body 1.

Moreover, in the image-pickup/detection unit 1100, the optical axis OA′of the image pickup lens 1101 is located on the optical axis OA2 of theface direction detection lens 1102. Furthermore, the two infrared LEDs1110 a and 1110 b are mounted on the flexible substrate 1110 in linesymmetry about the optical axis OA2 of the face direction detection lens1102. Thereby, the weight balance of the image-pickup/detection unit1100 is stabilized, which can prevent the image pickup lens 1101 and theface direction detection lens 1102 from inclining with respect to themedian plane. As a result, when wearing the camera body 1 of the secondembodiment, the user 1112 can pick up an image stably with the camerabody 1 concerned.

Moreover, since the optical axis OA2 of the face direction detectionlens 1102 and the optical axis OA1 of the image pickup lens 1101intersect on the median plane, the positions of the optical axes of theface direction detection lens 1102 and image pickup lens 1101 arematched in the lateral direction (X-direction in the drawing). Thereby,when the area of the image recording frame that is extracted from thesuperwide-angle image described in the first embodiment is determined,it becomes unnecessary to consider the deviation between the opticalaxes of the face direction detection lens 1102 and image pickup lens1101 in the lateral direction. As a result, the load of the calculationprocess for determining the extraction area can be reduced.

Next, details of the slide operation of the lens barrier 1111 in thesecond embodiment will be described. The image pickup lens 1101 in thesecond embodiment is the fish-eye lens as mentioned above, and its partprotrudes from the opening 1104 a of the front cover 1104 ahead(Z-direction in the drawing). Accordingly, if the lens barrier 1111should move in the vertical direction (Y-direction in the drawing)without moving in the front-back direction, the lens barrier 1111 mayinterfere with the protruded part of the image pickup lens 1101.

In order to avoid the interference, when the lens barrier 1111 moves inthe vertical direction, it moves also in the front-back direction in thesecond embodiment. Specifically, when the lens barrier 1111 movesdownwardly to cover the image pickup lens 1101, the lens barrier 1111moves ahead (FIG. 23A). Moreover, when the lens barrier 1111 movesupwardly along the surface of the front cover 1104, the lens barrier1111 moves back (FIG. 23B). At this time, the lens barrier 1111 retractsfrom the optical axis OA1 of the image pickup lens 1101. Interlockingbetween the movement in the vertical direction and the movement in thefront-back direction of the lens barrier 1111 is achieved by a camconnection mechanism (not shown) arranged between the lens barrier 1111and the front cover 1104. A cam locus formed in this cam connectionmechanism has a shape following a curved surface of the image pickuplens 1101. Thereby, the lens barrier 1111 does not interfere with thesurface of the image pickup lens 1101 and moves so as to trace thesurface concerned. It should be noted that lens barrier 1111 is manuallymoved by the user 1112 in the vertical direction. Moreover, a motor thatdrives the cam connection mechanism may be provided and the lens barrier1111 may be moved in the vertical direction by motor drive triggered byan operation of a button etc. by the user 1112.

Moreover, as mentioned above, when the lens barrier 1111 is movedupwardly and the image pickup lens 1101 is exposed to the outside, apart of the lens barrier 1111 protrudes from the upper end of the frontcover 1104 (FIG. 22B, FIG. 23B). The protruded part of the lens barrier1111 functions as a shading wall that reduces incidence of externallight like sunlight penetrated from the front of the user 1112 into theface direction detection lens 1102 through the upper cover 1109. Thisprevents lowering of the accuracy of detection of the observationdirection of the user 1112 by the observation direction detection unit.That is, the lens barrier 1111 performs two functions to cover andprotect the image pickup lens 1101 in the non-image-pickup state and toreduce the incidence of the external light into the face directiondetection lens 1102 in the image-pickup state, which contributes to theminiaturization of the camera body 1 and the simplification of theconfiguration.

Next, a third embodiment will be described. The third embodiment isbasically identical to the second embodiment in the configuration, butthe contour of the image-pickup/detection unit 1100 differs.Hereinafter, a component that has the same function and configuration asthe component already described in the second embodiment is indicated bythe same reference numeral, its description in the specification isomitted, and only a configuration peculiar to the third embodiment isdescribed.

FIG. 25 is a front view schematically showing the configuration of theimage-pickup/detection unit 1100 according to the third embodiment. Itshould be noted that the front cover 1104, back cover 1105, upper cover1109, and lens barrier 1111 are omitted in FIG. 25 . In the secondembodiment, the substrate 1103 is configured to extend in the verticaldirection (Y-direction in the drawing) when the user 1112 is equippedwith the camera body 1. As a result, when viewed from the front of theuser 1112, the image-pickup/detection unit 1100 also extends in thevertical direction and the contour of which the overall length in thevertical direction becomes longer than the overall length in the lateraldirection is exhibited.

In the meantime, the substrate 1103 is configured to extend in thelateral direction (X-direction in the drawing) when the user 1112 isequipped with the camera body 1 in the third embodiment. Thereby, whenviewed from the front of the user 1112, the image-pickup/detection unit1100 also extends in the lateral direction and the contour of which theoverall length in the lateral direction becomes longer than the overalllength in the vertical direction is exhibited. As a result, weightbalance in the lateral direction of the image-pickup/detection unit 1100is easily maintained, and the image-pickup/detection unit 1100 isstabilized. That is, the optical axis OA2 of the face directiondetection lens 110 does not depart greatly from the jaw of the user1112, and the observation direction of the user 1112 can be easilydetected.

Next, a fourth embodiment will be described. The fourth embodiment isbasically identical to the second embodiment in the configuration but isdifferent from the second embodiment in that the observation-directiondetection unit including the face direction detection lens 1102 isconstituted to be rotatable with respect to the image pickup unitincluding the image pickup lens 1101. Hereinafter, a component that hasthe same function and configuration as the component already describedin the second embodiment is indicated by the same reference numeral, itsdescription in the specification is omitted, and only a configurationpeculiar to the fourth embodiment is described.

FIG. 26 is an exploded perspective view schematically showing aconfiguration of the image-pickup/detection unit 1100 according to thefourth embodiment. FIG. 27 is a sectional view along the optical axisOA1 and optical axis OA2 schematically showing an internal configurationof the image-pickup/detection unit 1100 according to the fourthembodiment.

As shown in FIG. 26 and FIG. 27 , the image-pickup/detection unit 1100according to the fourth embodiment has a front cover 1114 and back cover1115 that have different shapes from the front cover 1104 and back cover1105 instead of them. Furthermore, the image-pickup/detection unit 1100has a face-detection-lens holding frame (rotating member) 1106 thatrotates the observation-direction detection unit. Theface-detection-lens holding frame 1106 is a housing and stores the facedirection detection lens 1102, the infrared LEDs 1110 a, 1110 b, and theflexible substrate 1110 inside. Moreover, the upper side of theface-detection-lens holding frame 1106 opens, and the upper cover 1109is attached to the opening.

Side walls of the front cover 1114 in the lateral direction (X-directionin the drawing) protrude upwardly (Y-direction in the drawing) so as toform support parts 1114 c and 1114 d, respectively. It should be notedthat the back cover 1115 also has similar support parts. Theface-detection-lens holding frame 1106 is arranged between the supportparts 1114 c and 1114 d.

Holes 1114 a and 1114 b are respectively provided in the support parts1114 c and 1114 d, and screw holes 1106 a and 1106 b are provided in theface-detection-lens holding frame 1106 so as to correspond to the holes1114 a and 1114 b, respectively. Stepped screws 1107 a and 1107 b arescrewed into the screw holes 1106 a and 1106 b through the holes 1114 aand 1114 b of the support parts 1114 c and 1114 d. At this time, sinceshafts of the stepped screws 1107 a and 1107 b are loosely fitted to theholes 1114 a and 1114 b, the face-detection-lens holding frame 1106 isaxially supported by the stepped screws 1107 a and 1107 b with respectto the support parts 1114 c and 1114 d of the front cover 1114 as aresult. Thereby, the face-detection-lens holding frame 1106 becomesrotatable with respect to the front cover 1104 around the shafts of thestepped screws 1107 a and 1107 b. Specifically, since a rotation centerRC of the face-detection-lens holding frame 1106 that is an axial centerof the screw holes 1106 a and 1106 b perpendicularly intersects with theoptical axis OA1 of the image pickup lens 1101, the face-detection-lensholding frame 1106 rotates in a tilting direction against the imagepickup lens 1101.

Moreover, washers 1108 a and 1108 b for adjusting torque are arrangedbetween the stepped screws 1107 a and 1107 b and the holes 1114 a and1114 b, and the face-detection-lens holding frame 1106 is rotated withproper torque.

Incidentally, when the user 1112 wears the camera body 1 correctly, theoptical axis OA1 of the image pickup lens 1101 matches the horizontaldirection (Z-direction in FIG. 26 and FIG. 27 ). However, when the user1112 wears the camera body 1 incorrectly and the camera body 1 inclinesfrom the user 1112, the optical axis OA1 does not match the horizontaldirection and may incline so as to tilt from the horizontal direction(FIG. 27 ). In such a case, when the optical axis OA2 of the facedirection detection lens 1102 always intersects perpendicularly with theoptical axis OA1 as with the second embodiment or third embodiment, theoptical axis OA2 does not directly face the jaw of the user 1112, andthe face direction detection lens 1102 does not point to the jaw.

In response to this, in the fourth embodiment, the face-detection-lensholding frame 1106 is constituted so as to be rotatable in the tiltingdirection against the front cover 1114 and back cover 1115 as mentionedabove. Thereby, the user 1112 can make the face direction detection lens1102 point to the jaw irrespective of the direction of the optical axisOA1 by manually rotating the face-detection-lens holding frame 1106 inthe tilting direction with respect to the front cover 1114.

Moreover, in the fourth embodiment, an image sensor 1102 a of theobservation-direction detection unit is arranged so as to be located atthe same height as the rotation center RC of the face-detection-lensholding frame 1106. Thereby, even if the face direction detection lens1102 rotates with respect to the image pickup lens 1101, the imagepickup field of the face direction detection lens 1102 does not vary.

According to the above configuration, even if the user 1112 isincorrectly equipped with the camera body 1, the observation directionof the user 1112 is certainly detectable in the fourth embodiment.

It should be noted that the user 1112 adjusts the rotation angle bymanually rotating the face-detection-lens holding frame 1106 in thefourth embodiment. However, the image-pickup/detection unit 1100 may beprovided with a driving source, for example a motor, that drives theface-detection-lens holding frame 1106 so as to automatically adjust therotation angle of the face-detection-lens holding frame 1106 using themotor.

Moreover, the face-detection-lens holding frame 1106 is rotated in thetilting direction with respect to the image pickup lens 1101 in thefourth embodiment. However, the face-detection-lens holding frame 1106may be rotated in a rolling direction with respect to the image pickuplens 1101 by locating the rotation center RC of the face-detection-lensholding frame 1106 on the median plane. Furthermore, theface-detection-lens holding frame may be constituted as double structureso that the face direction detection lens 1102 may be rotated in boththe tilting direction and rolling direction with respect to the imagepickup lens 1101.

Next, a fifth embodiment will be described. The fifth embodiment isbasically identical to the third embodiment in the configuration but isdifferent from the third embodiment in that two image pickup lenses areprovided. Hereinafter, a component that has the same function andconfiguration as the component already described in the third embodimentis indicated by the same reference numeral, its description in thespecification is omitted, and only a configuration peculiar to the fifthembodiment is described.

FIG. 28 is a front view schematically showing the configuration of theimage-pickup/detection unit 1100 according to the fifth embodiment. Itshould be noted that the front cover 1104, back cover 1105, upper cover1109, and lens barrier 1111 are omitted in FIG. 28 as with FIG. 25 . Inthe fifth embodiment, the substrate 1103 is configured to extend in thelateral direction (X-direction in the drawing) as with the thirdembodiment.

As shown in FIG. 28 , the image-pickup/detection unit 1100 has a rightimage pickup lens 1101 a and a left image pickup lens 1101 b. In theimage-pickup/detection unit 1100 of the fifth embodiment, when thecamera body 1 is viewed from the front, an optical axis OA3 of the rightimage pickup lens 1101 a and an optical axis OA4 of the left imagepickup lens 1101 b are not respectively located in the median planealong which the optical axis OA2 of the face direction detection lens1102 passes. Then, a center of gravity CG of the right image pickup lens1101 a and the left image pickup lens 1101 b is located in the medianplane.

Thereby, weight balance of the image-pickup/detection unit 1100 in thelateral direction is easily maintained, which can reduce inclination ofthe image-pickup/detection unit 1100 with respect to the user 1112 whenthe user 1112 wears the camera body 1. As a result, it can bestabilized, and can be made to be able to point to the face directiondetection lens 1102 to a user's 1112 jaw, and the detection accuracy ofa user's 1112 observation direction can be prevented from getting worse.

Although the image-pickup/detection unit 1100 of the fifth embodimenthas the two image pickup lenses 1101 a and 1101 b, theimage-pickup/detection unit 1100 may have three or more image pickuplenses. However, in any case, the center of gravity of a plurality ofimage pickup lenses needs to be located in the median plane along whichthe optical axis OA2 of the face direction detection lens 1102 passes.

Next, a sixth embodiment will be described. The sixth embodiment isbasically identical to the second embodiment in the configuration, andan image-pickup/detection unit 2100 is provided in the camera body 1instead of the image-pickup/detection unit 1100. Hereinafter, acomponent that has the same function and configuration as the componentalready described in the first embodiment and second embodiment isindicated by the same name and reference numeral, its description in thespecification is omitted, and only a configuration peculiar to the sixthembodiment is described.

FIG. 29 is an exploded perspective view schematically showing theconfiguration of the image-pickup/detection unit 2100 according to thesixth embodiment. As shown in FIG. 29 , the image-pickup/detection unit2100 has an image pickup unit for picking up an object and anobservation-direction detection unit for detecting an observationdirection of a user. The image pickup unit is provided with an imagepickup lens 2101 and an image sensor 2103 a. The observation-directiondetection unit is provided with a face direction detection lens 2102 andan image sensor (not shown). The image pickup lens 2101 is an imagepickup optical component that has an image pickup optical axis OA5. Theface direction detection lens 2102 is an observation-direction-detectionoptical component that has a detection optical axis OA6. The imagepickup lens 2101 and the image sensor 2103 a are supported by asubstrate 2103. Moreover, the face direction detection lens 2102 and theimage sensor (not shown) are supported by anobservation-direction-detection flexible substrate 2110. Furthermore,the image-pickup/detection unit 2100 has a front cover 2104, a backcover 2105, and an observation-direction-detection window 2109.

The observation-direction-detection flexible substrate 2110 is connectedto the substrate 2103. Moreover, infrared LEDs 2110 a and 2110 b, whichirradiate each part of a user's face with infrared light, and a circuitfor lighting them are mounted on the observing-direction-detectionflexible substrate 2110. Although the arrangement number of the infraredLEDs is two in this embodiment, it is not limited to this.

The front cover 2104 is a box-shaped member, and an opening 2104 c isformed at a place that faces the image pickup lens 2101 in the frontcover 2104. In this embodiment, the image pickup lens 2101 is a fish-eyelens and its part protrudes from the front cover 2104 to the outsidethrough the opening 2104 c. An observation-direction-detection window2109 is fitted in a top plate of the front cover 2104. Theobservation-direction-detection window 2109 is a transparent platemember and is arranged above the face direction detection lens 2102 andthe infrared LEDs 2110 a and 2110 b. Moreover, theobservation-direction-detection window 2109 overlaps with the facedirection detection lens 2102, infrared LEDs 2110 a and 2110 b, when theobservation-direction-detection window 2109 is viewed from the upperside. The back cover 2105 is a box-shaped member that covers from theback side of the front cover 2104. Then, the image pickup lens 2101,face direction detection lens 2102, substrate 2103,observation-direction-detection flexible substrate 2110, etc. are storedand arranged in a space surrounded by the front cover 2104, back cover2105, and observation-direction-detection window 2109. In thisembodiment, the space surrounded by the front cover 2104, back cover2105, observation-direction-detection window 2109, and the unitsarranged in the space surrounded by them constitute a main body 2140 ofthe image-pickup/detection unit 2100.

As shown in FIG. 29 , the image-pickup/detection unit 2100 has a firstbutton 2121 (first recording start member), a second button 2122 (secondrecording start member), and an operation-unit flexible substrate 2120.Although the first button 2121 and the second button 2122 are arrangedat mutually different positions, both of the buttons function as arecording start member that the user designates to start recording animage. Moreover, both of the first button 2121 and the second button2122 are operated by a press operation. The operation-unit flexiblesubstrate 2120 is connected to the substrate 2103 and has a first switch2120 a that detects a press operation to the first button 2121 and asecond switch 2120 b that detects a press operation to the second button2122.

FIG. 30 is a view showing a state where a user 2112 wears the camerabody 1 equipped with the image-pickup/detection unit 2100 and isperforming an image pickup start operation. In FIG. 30 , the components,such as the connection members 80 and the battery unit 90 s of thecamera body 1 other than image-pickup/detection unit 2100 are not shown.FIG. 31 is a front view showing the image-pickup/detection unit 2100when the front cover 2104 is removed.

As shown in FIG. 30 , the user 2112 wears the camera body 1 so that theimage-pickup/detection unit 2100 will be located on the median plane ofthe user 2112 and under a jaw. When the camera body 1 is viewed from thefront of the user 2112, the face direction detection lens 2102 in theimage-pickup/detection unit 2100 is located on the median plane of theuser and under the jaw of the user. Moreover, a detection optical axisOA6 of the face direction detection lens 2102 approximately matches themedian plane (shown by an alternate long and short dash line in thedrawing) of the user 2112. Since the user 2112 and the face directiondetection lens 2102 satisfy such positional relationship, the facedirection detection lens 2102 points to the jaw of the user 2112.Thereby, the face direction detection can be performed easily and theaccuracy of the face direction detection can be raised.

An image pickup optical axis OA5 of the image pickup lens 2101 does notmatch the median plane of the user 2112, i.e., is not located on themedian plane. The image pickup optical axis OA5 and the detectionoptical axis OA6 are in a twisted positional relation where they do notintersect mutually. Since the image pickup optical axis OA5 shifts fromthe median plane of the user 2112, parallax in the lateral directionoccurs. However, since the image pickup lens 2101 is the fish-eye lensas mentioned above, influence of the parallax in picking up an image isslight.

Moreover, since the image pickup optical axis OA5 and the detectionoptical axis OA6 are in the twisted positional relation, the imagepickup lens 2101 and the face direction detection lens 2102 are notstacked in a height direction of the image-pickup/detection unit 2100,i.e., an overlapped arrangement is avoidable. This can reduce the sizeof the image-pickup/detection unit 2100 in the height direction and canminiaturize the image-pickup/detection unit 2100. Moreover, the imagepickup lens 2101 can be approached to height of an eye line of the user2112 as much as possible. Thereby, the image recorded by the camera body1 can be approximated to the image that the user 2112 is looking at.

As shown in FIG. 31 , the first button 2121 is protrusively arranged ona lower part of a right side surface (side wall) 2104 a of the frontcover 2104 of the main body 2140 a when viewed from the front of theuser 2112 who wears the camera body 1. The second button 2122 isarranged at the side surface opposite to the first button 2121. That is,the second button 2122 is protrusively arranged on a lower part of aleft side surface (side wall) 2104 b of the front cover 2104.

In this embodiment, when both the first button 2121 and second button2122 are press-operated simultaneously, the simultaneous operations aredetected by the first switch 2120 a and second switch 2120 b. Then, theoverall control CPU 101 starts recording an image, when the concurrentoperations are detected by the first switch 2120 a and second switch2120 b. Such a configuration prevents unintentional start of the imagerecording, even if one of the first button 2121 and second button 2122is erroneously press-operated, for example.

Moreover, the first button 2121 and second button 2122 are arrangedmutually opposite sides, i.e., right and left sides, of the main body2140 as shown in FIG. 30 . Thereby, the operations of the first button2121 and the second button 2122 by the user 2112 become available whilepinching the image-pickup/detection unit 2100 between a thumb 2130 andan index finger 2131 from its bottom (hereinafter referred to as“pinching from the bottom”). Such operations are preferable for thesimultaneous operations of the buttons.

Moreover, the user 2112 who wears the image pickup apparatus (camerabody 1) often performs a recording start operation suddenly when animage pickup chance appears in a view of the user 2112. Accordingly, thefirst button 2121 and second button 2122 are required to be easilyoperated by any of right and left hands. Moreover, the user 2112 canoperate the buttons by any of right and left hands with a similaroperational feeling by the pinching from the bottom mentioned above.And, reflection of a hand or a finger on a picked-up image at arecording start timing is reduced. Furthermore, even in a case where theimage-pickup/detection unit 2100 inclines or rotates to the user 2112,the user 2112 can naturally return the image-pickup/detection unit 2100to a correct position by the pinching from the bottom.

Operation force (pressure force) F2121 in press-operating the firstbutton 2121 mainly depends on torque needed to operate the first switch2120 a (see FIG. 31 ). Operation force (pressure force) F2122 inpress-operating the second button 2122 mainly depends on torque neededto operate the second switch 2120 b (see FIG. 31 ).

In the image-pickup/detection unit 2100, the torques of the first andsecond switches 2120 a and 2120 b are set up so that the operation forceF2121 will become larger than the operation force F2122. When thebuttons are simultaneously operated, such a size relationship of theforces gives a role of a support side to the second button 2122 of whichthe operation force F2122 is small and give a role of a pushing side tothe first button 2121 of which the operation force F2121 is large. As aresult, movement of the thumb 2130 put on the second button 2122 can bereduced as much as possible, and accordingly the simultaneous operationsbecome easy. In order to enable such simultaneous operations, it ispreferable that the operation force F1212 becomes a double or more ofthe operation force F2121, for example. However, the relation is notlimited to this.

Although the size relationship between the operation force F2121 and theoperation force F2122 is set up by the balance of the torques requiredfor operating the switches in the above-mentioned example, the settingis not limited to this. For example, the size relationship between theoperation forces may be set up by elastic members, such as springs.

It is preferable that a distance L (see FIG. 31 ) between the firstbutton 2121 and the second button 2122 in a state where the first button2121 and the second button 2122 are projected at the maximum (i.e., astate before operations) falls within a range that is 50 mm or more andis 200 mm or less, for example. If the distance L is less than 50 mm,some users 2112 may be difficult to perform the pinching from the bottommentioned above. Moreover, if the distance L is more than 200 mm ormore, a user whose hand is small may be difficult to perform thepinching from the bottom mentioned above by a single hand.

Moreover, when the image-pickup/detection unit 2100 is miniaturizedwhile satisfying the distance L being 50 mm or more, it is preferablethat the image-pickup/detection unit 2100 has a laterally wider shape.Accordingly, the suppression of the size of the image-pickup/detectionunit 2100 in the height direction by arranging the image pickup lens2101 and face direction detection lens 2102 so as to be mutuallydeviated in the lateral direction contributes to miniaturization of theimage-pickup/detection unit 2100.

Next, a seventh embodiment will be described using FIG. 32A through FIG.32C. The seventh embodiment is basically identical to the sixthembodiment in the configuration. Hereinafter, a component that has thesame function and configuration as the component already described inthe sixth embodiment is indicated by the same reference numeral, itsdescription in the specification is omitted, and only a configurationpeculiar to the seventh embodiment is described.

FIG. 32A through FIG. 32C are views showing variations of shapes of afirst button 2221 and a second button 2222 according to the seventhembodiment.

In this embodiment, an image-pickup/detection unit 2200 has the firstbutton 2221 and second button 2222 that are mutually different in ashape (including size). In a configuration shown in FIG. 32A, the firstbutton 2221 and second button 2222 are mutually different in a size. Adiameter Φd2221 of the first button 2221 is smaller than a diameterΦd2222 of the second button 2222. In a configuration shown in FIG. 32B,the first button 2221 and second button 2222 are mutually different in aprotrusion amount. A protrusion amount Q2221 of the first button 2221 issmaller than a protrusion amount Q2222 of the second button 2222. In aconfiguration shown in FIG. 32C, the first button 2221 and second button2222 are mutually different in a state of a top surface A plurality ofdepressions and projections are formed on a top surface 2221 a of thefirst button 2221, and a top surface 2222 a of the second button 2222 issmoother than the top surface 2221 a.

When the shapes of the first button 2221 and second button 2222 aremutually different in the shape, touch feelings of the buttons aredifferent, respectively. Thereby, when the user 2112 tries to operatethe buttons by any of right and left hand suddenly, the user 2112 canpromptly distinguish the high operation force button and low operationforce button, which facilitates reduction of an operation shake.

Although the first button 2221 and second button 2222 are constituted sothat the shapes are mutually different in this embodiment, they may beconstituted so that rigidities or flexibilities are mutually different,for example. In such a case, the touch feelings of the buttons aredifferent, respectively.

Next, an eighth embodiment will be described using FIG. 33 . The eighthembodiment is basically identical to the sixth embodiment in theconfiguration. Hereinafter, a component that has the same function andconfiguration as the component already described in the sixth embodimentis indicated by the same reference numeral, its description in thespecification is omitted, and only a configuration peculiar to theeighth embodiment is described.

FIG. 33 is a view showing a state where the user 2112 wears the camerabody 1 equipped with the image-pickup/detection unit 2300 and isperforming an image pickup start operation.

The image-pickup/detection unit 2300 of this embodiment omits the secondbutton 2122 and the second switch 2120 b as compared with theimage-pickup/detection unit 2100 of the sixth embodiment. Moreover, theimage-pickup/detection unit 2300 has a finger placement member 2340 (asupport member) instead of the second button 2122. The finger placementmember 2340 is arranged at the side surface opposite to the first button2121 as with the second button 2122. That is, the finger placementmember 2340 is arranged on the lower part of the left side surface 2104b.

When performing the recording start operation, the user 2112 places(attaches) the thumb 2130 on the finger placement member 2340 to holdthe main body 2140. This stabilizes the press operation of the firstbutton 2121. Although the finger placement member 2340 shown in FIG. 33has a flat shape and is protruded from the left side surface 2104 b, itmay be formed in the same surface as the left side surface 2104 b.Moreover, although the example that the user 2112 places the finger onthe finger placement member 2340 to hold the main body 2340 in pressingthe first button 2121 is shown in this embodiment, the holding mechanismis not limited to this. For example, the user 2112 may be equipped witha holding member that prevents movement of the finger placement member2340 in pressing the first button 2121 on the user's body. In such acase, the main body 2140 can be held by abutting or fitting the fingerplacement member 2340 to the holding member.

OTHER EMBODIMENTS

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Applications No.2021-126103, filed Jul. 30, 2021, and No. 2022-038689, filed Mar. 11,2022, which are hereby incorporated by reference herein in theirentireties.

What is claimed is:
 1. An image pickup apparatus comprising: anobservation direction detection unit that is adapted to be worn on abody part other than a head of a user and that is configured to detectan observation direction of the user; at least one image pickup unitthat is adapted to be worn on the body part of the user and that isconfigured to pick up an image; a recording direction determination unitconfigured to determine a recording direction in accordance with theobservation direction detected; and an image recording unit configuredto record an image corresponding to the recording direction determinedfrom among an image picked up by the at least one image pickup unit,wherein the observation direction detection unit and the image pickupunit are located on a median plane of the user and under a jaw of theuser in a worn state where the user wears the image pickup apparatus,and wherein the observation direction detection unit is located at anearer position to the jaw than the at least one image pickup unit inthe worn state.
 2. The image pickup apparatus according to claim 1,wherein the at least one image pickup unit comprises a plurality ofimage pickup units, wherein a center of gravity of the plurality ofimage pickup units is located on the median plane.
 3. The image pickupapparatus according to claim 1, further comprising a rotating memberthat rotates the observation direction detection unit with respect tothe at least one image pickup unit.
 4. The image pickup apparatusaccording to claim 3, wherein the rotating member rotates theobservation direction detection unit in at least one of a rollingdirection and a tilting direction with respect to the at least one imagepickup unit.
 5. The image pickup apparatus according to claim 1, furthercomprising a protection member that protects the at least one imagepickup unit, wherein the protection member moves in a direction thatintersects perpendicularly with an optical axis of the at least oneimage pickup unit so as to vary between a first state located on theoptical axis and a second state retracted from the optical axis.
 6. Theimage pickup apparatus according to claim 5, wherein the protectionmember moves in a direction toward the observation direction detectionunit when moving from the first state to the second state.
 7. The imagepickup apparatus according to claim 6, wherein a part of the protectionmember protrudes toward the jaw in the second state.
 8. The image pickupapparatus according to claim 1, further comprising at least two lightemission members, wherein the at least two light emission members arearranged in line symmetry about an optical axis of the observationdirection detection unit.
 9. The image pickup apparatus according toclaim 1, wherein a contour of which an overall length in a lateraldirection becomes longer than an overall length in a vertical directionwhen viewed from a front of the user is exhibited in the worn state. 10.An image pickup apparatus comprising: an observation direction detectionunit that is adapted to be worn on a body part other than a head of auser and that is configured to detect an observation direction of theuser; and an image pickup unit configured to pick up an image, whereinthe observation direction detection unit includes anobservation-direction-detection optical component that has a detectionoptical axis, wherein the image pickup unit includes an image pickupoptical component that has an image pickup optical axis, and wherein thedetection optical axis and the image pickup optical axis are in atwisted positional relation where they do not intersect mutually. 11.The image pickup apparatus according to claim 10, wherein theobservation direction detection unit is located on a median plane of theuser and under a jaw of the user in a worn state where the user wearsthe image pickup apparatus, and wherein the image pickup unit is notlocated on the median plane in the worn state.
 12. An image pickupapparatus comprising: a main body that includes an observation directiondetection unit that is adapted to be worn on a body part other than ahead of a user and that is configured to detect an observation directionof the user, and an image pickup unit configured to pick up an image; afirst recording start member that is provided in a side surface of themain body and that is used by the user in designating to start recordingan image; and a second recording start member that is provided in a sidesurface of the main body at an opposite side of the first recordingstart member and that is used by the user in designating to startrecording an image, wherein the image pickup unit starts recording theimage in a case where both the first recording start member and thesecond recording start member are operated simultaneously.
 13. The imagepickup apparatus according to claim 12, wherein operation force requiredto operate the first recording start member is larger than operationforce required to operate the second recording start member.
 14. Theimage pickup apparatus according to claim 12, wherein a shape of thefirst recording start member differs from a shape of the secondrecording start member.
 15. The image pickup apparatus according toclaim 12, wherein a distance between the first recording start memberand the second recording start member is 50 mm or more.
 16. The imagepickup apparatus according to claim 12, wherein a distance between thefirst recording start member and the second recording start member is200 mm or less.
 17. The image pickup apparatus according to claim 12,wherein a distance between the first recording start member and thesecond recording start member falls within a range that is 50 mm or moreand is 200 mm or less.
 18. An image pickup apparatus comprising: a mainbody that includes an observation direction detection unit that isadapted to be worn on a body part other than a head of a user and thatis configured to detect an observation direction of the user, and animage pickup unit that picks up an image; a recording start member thatis used by the user in designating to start recording the image; and asupport member that holds the main body in operating the recording startmember, wherein the recording start member is provided in a side surfaceof the main body, and wherein the support member is arranged on a sidesurface of the main body at an opposite side of the recording startmember.